Actions
Bug #42918
closedmemory corruption and lockups with I-Object
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
http://pulpito.ceph.com/dis-2019-11-20_20:35:04-krbd-master-wip-krbd-readonly-basic-mira/4526411
2019-11-20T21:22:45.444 INFO:tasks.workunit.client.0.mira034.stderr:+ rbd snap create testimg1 --snap=snap1 2019-11-20T21:22:45.544 INFO:tasks.workunit.client.0.mira034.stderr:2019-11-20T21:22:45.536+0000 7f9ae31e6700 0 -- 172.21.4.108:0/2432388899 >> [v2:172.21.6.136:6800/29209,v1:172.21.6.136:6801/29209] conn(0x7f9ac40170d0 msgr2=0x7f9ac4019540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-11-20T21:22:45.634 INFO:tasks.workunit.client.0.mira034.stderr:2019-11-20T21:22:45.627+0000 7f9ae39e7700 0 -- 172.21.4.108:0/2432388899 >> [v2:172.21.9.138:6808/30243,v1:172.21.9.138:6810/30243] conn(0x563809c7ca40 msgr2=0x563809c7ee40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-11-20T21:22:45.645 INFO:tasks.workunit.client.0.mira034.stderr:2019-11-20T21:22:45.637+0000 7f9ae39e7700 0 -- 172.21.4.108:0/2432388899 >> [v2:172.21.9.138:6808/30243,v1:172.21.9.138:6810/30243] conn(0x7f9ac401e230 msgr2=0x7f9ac40206d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-11-20T21:22:45.651 INFO:tasks.workunit.client.0.mira034.stderr:*** Caught signal (Segmentation fault) ** 2019-11-20T21:22:45.651 INFO:tasks.workunit.client.0.mira034.stderr: in thread 7f9ae21e4700 thread_name:rbd 2019-11-20T21:22:45.652 INFO:tasks.workunit.client.0.mira034.stderr: ceph version 15.0.0-7426-g15b37e0 (15b37e0f4e7f6930b0d185998c098a2a36d5a6d7) octopus (dev) 2019-11-20T21:22:45.652 INFO:tasks.workunit.client.0.mira034.stderr: 1: (()+0xf5d0) [0x7f9ae922c5d0] 2019-11-20T21:22:45.653 INFO:tasks.workunit.client.0.mira034.stderr: 2: (()+0xa75f4) [0x7f9af32005f4] 2019-11-20T21:22:45.653 INFO:tasks.workunit.client.0.mira034.stderr: 3: (()+0x9fab3) [0x7f9af31f8ab3] 2019-11-20T21:22:45.653 INFO:tasks.workunit.client.0.mira034.stderr: 4: (()+0xcc85d) [0x7f9af322585d] 2019-11-20T21:22:45.653 INFO:tasks.workunit.client.0.mira034.stderr: 5: (()+0x10cf63) [0x7f9af3265f63] 2019-11-20T21:22:45.653 INFO:tasks.workunit.client.0.mira034.stderr: 6: (()+0xb7265) [0x7f9af3210265] 2019-11-20T21:22:45.653 INFO:tasks.workunit.client.0.mira034.stderr: 7: (()+0xbce81) [0x7f9af3215e81] 2019-11-20T21:22:45.653 INFO:tasks.workunit.client.0.mira034.stderr: 8: (()+0x84040f) [0x7f9aea0da40f] 2019-11-20T21:22:45.654 INFO:tasks.workunit.client.0.mira034.stderr: 9: (()+0x7dd5) [0x7f9ae9224dd5] 2019-11-20T21:22:45.654 INFO:tasks.workunit.client.0.mira034.stderr: 10: (clone()+0x6d) [0x7f9ae78b002d] 2019-11-20T21:22:45.657 INFO:tasks.workunit.client.0.mira034.stderr:2019-11-20T21:22:45.645+0000 7f9ae21e4700 -1 *** Caught signal (Segmentation fault) ** 2019-11-20T21:22:45.657 INFO:tasks.workunit.client.0.mira034.stderr: in thread 7f9ae21e4700 thread_name:rbd
http://pulpito.ceph.com/dis-2019-11-20_20:35:04-krbd-master-wip-krbd-readonly-basic-mira/4526412
2019-11-20T21:26:49.467 INFO:tasks.workunit.client.0.mira109.stderr:+ rbd rm testimg1 2019-11-20T21:26:49.784 INFO:tasks.workunit.client.0.mira109.stderr:2019-11-20T21:26:49.778+0000 7f3442a0d700 0 -- 172.21.9.140:0/2792304097 >> [v2:172.21.6.116:6800/29304,v1:172.21.6.116:6801/29304] conn(0x7f34300082c0 msgr2=0x7f343000a790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-11-20T21:26:49.801 INFO:tasks.workunit.client.0.mira109.stderr: Removing image: 5% complete... Removing image: 10% complete... Removing image: 15% complete... Removing image: 21% complete... Removing image: 26% complete... Removing image: 31% complete... Removing image: 36% complete... Removing image: 42% complete... Removing image: 47% complete... Removing image: 52% complete... Removing image: 57% complete... Removing image: 63% complete... Removing image: 68% complete... Removing image: 73% complete... Removing image: 78% complete... Removing image: 84% complete...2019-11-20T21:26:49.794+0000 7f3442a0d700 0 -- 172.21.9.140:0/2792304097 >> [v2:172.21.6.116:6800/29304,v1:172.21.6.116:6801/29304] conn(0x7f342401e300 msgr2=0x7f34240207d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-11-20T21:26:49.813 INFO:tasks.workunit.client.0.mira109.stderr: Removing image: 89% complete... Removing image: 94% complete...*** Error in `rbd': malloc(): memory corruption: 0x00007f3420000080 *** 2019-11-20T21:26:49.813 INFO:tasks.workunit.client.0.mira109.stderr:======= Backtrace: ========= 2019-11-20T21:26:49.813 INFO:tasks.workunit.client.0.mira109.stderr:/lib64/libc.so.6(+0x82e16)[0x7f344785ce16] 2019-11-20T21:26:49.813 INFO:tasks.workunit.client.0.mira109.stderr:/lib64/libc.so.6(__libc_malloc+0x4c)[0x7f344785f32c] 2019-11-20T21:26:49.813 INFO:tasks.workunit.client.0.mira109.stderr:/lib64/libstdc++.so.6(_Znwm+0x1d)[0x7f344811decd] 2019-11-20T21:26:49.813 INFO:tasks.workunit.client.0.mira109.stderr:/lib64/librbd.so.1(+0x20b497)[0x7f345370a497] 2019-11-20T21:26:49.814 INFO:tasks.workunit.client.0.mira109.stderr:/lib64/librbd.so.1(+0x20b867)[0x7f345370a867] 2019-11-20T21:26:49.814 INFO:tasks.workunit.client.0.mira109.stderr:/lib64/librbd.so.1(+0xcf969)[0x7f34535ce969] 2019-11-20T21:26:49.816 INFO:tasks.workunit.client.0.mira109.stderr:/usr/lib64/ceph/libceph-common.so.2(_ZN10ThreadPool6workerEPNS_10WorkThreadE+0xd70)[0x7f3449c29ff0] 2019-11-20T21:26:49.816 INFO:tasks.workunit.client.0.mira109.stderr:/usr/lib64/ceph/libceph-common.so.2(_ZN10ThreadPool10WorkThread5entryEv+0x11)[0x7f3449c2a801] 2019-11-20T21:26:49.816 INFO:tasks.workunit.client.0.mira109.stderr:/lib64/libpthread.so.0(+0x7dd5)[0x7f344924cdd5] 2019-11-20T21:26:49.816 INFO:tasks.workunit.client.0.mira109.stderr:/lib64/libc.so.6(clone+0x6d)[0x7f34478d802d] 2019-11-20T21:26:49.816 INFO:tasks.workunit.client.0.mira109.stderr:======= Memory map: ======== 2019-11-20T21:26:49.817 INFO:tasks.workunit.client.0.mira109.stderr:55b861035000-55b86130f000 r-xp 00000000 08:01 75658 /usr/bin/rbd 2019-11-20T21:26:49.817 INFO:tasks.workunit.client.0.mira109.stderr:55b86150f000-55b861519000 r--p 002da000 08:01 75658 /usr/bin/rbd 2019-11-20T21:26:49.817 INFO:tasks.workunit.client.0.mira109.stderr:55b861519000-55b86151b000 rw-p 002e4000 08:01 75658 /usr/bin/rbd 2019-11-20T21:26:49.817 INFO:tasks.workunit.client.0.mira109.stderr:55b86151b000-55b861528000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.817 INFO:tasks.workunit.client.0.mira109.stderr:55b861655000-55b8618cc000 rw-p 00000000 00:00 0 [heap] 2019-11-20T21:26:49.817 INFO:tasks.workunit.client.0.mira109.stderr:7f3414000000-7f3414021000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.817 INFO:tasks.workunit.client.0.mira109.stderr:7f3414021000-7f3418000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.818 INFO:tasks.workunit.client.0.mira109.stderr:7f3418000000-7f3418021000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.818 INFO:tasks.workunit.client.0.mira109.stderr:7f3418021000-7f341c000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.818 INFO:tasks.workunit.client.0.mira109.stderr:7f341c000000-7f341c059000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.818 INFO:tasks.workunit.client.0.mira109.stderr:7f341c059000-7f3420000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.818 INFO:tasks.workunit.client.0.mira109.stderr:7f3420000000-7f3420021000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.818 INFO:tasks.workunit.client.0.mira109.stderr:7f3420021000-7f3424000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.818 INFO:tasks.workunit.client.0.mira109.stderr:7f3424000000-7f3424026000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.819 INFO:tasks.workunit.client.0.mira109.stderr:7f3424026000-7f3428000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.819 INFO:tasks.workunit.client.0.mira109.stderr:7f34287f9000-7f34287fa000 ---p 00000000 00:00 0 2019-11-20T21:26:49.819 INFO:tasks.workunit.client.0.mira109.stderr:7f34287fa000-7f3428ffa000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.819 INFO:tasks.workunit.client.0.mira109.stderr:7f3428ffa000-7f3428ffb000 ---p 00000000 00:00 0 2019-11-20T21:26:49.819 INFO:tasks.workunit.client.0.mira109.stderr:7f3428ffb000-7f34297fb000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.819 INFO:tasks.workunit.client.0.mira109.stderr:7f34297fb000-7f34297fc000 ---p 00000000 00:00 0 2019-11-20T21:26:49.819 INFO:tasks.workunit.client.0.mira109.stderr:7f34297fc000-7f3429ffc000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.820 INFO:tasks.workunit.client.0.mira109.stderr:7f3429ffc000-7f3429ffd000 ---p 00000000 00:00 0 2019-11-20T21:26:49.820 INFO:tasks.workunit.client.0.mira109.stderr:7f3429ffd000-7f342a7fd000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.820 INFO:tasks.workunit.client.0.mira109.stderr:7f342a7fd000-7f342a7fe000 ---p 00000000 00:00 0 2019-11-20T21:26:49.820 INFO:tasks.workunit.client.0.mira109.stderr:7f342a7fe000-7f342affe000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.820 INFO:tasks.workunit.client.0.mira109.stderr:7f342affe000-7f342afff000 ---p 00000000 00:00 0 2019-11-20T21:26:49.821 INFO:tasks.workunit.client.0.mira109.stderr:7f342afff000-7f342b7ff000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.821 INFO:tasks.workunit.client.0.mira109.stderr:7f342b7ff000-7f342b800000 ---p 00000000 00:00 0 2019-11-20T21:26:49.821 INFO:tasks.workunit.client.0.mira109.stderr:7f342b800000-7f342c000000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.821 INFO:tasks.workunit.client.0.mira109.stderr:7f342c000000-7f342c021000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.821 INFO:tasks.workunit.client.0.mira109.stderr:7f342c021000-7f3430000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.821 INFO:tasks.workunit.client.0.mira109.stderr:7f3430000000-7f3430021000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.821 INFO:tasks.workunit.client.0.mira109.stderr:7f3430021000-7f3434000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.822 INFO:tasks.workunit.client.0.mira109.stderr:7f3434000000-7f3434021000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.822 INFO:tasks.workunit.client.0.mira109.stderr:7f3434021000-7f3438000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.822 INFO:tasks.workunit.client.0.mira109.stderr:7f3438000000-7f3438035000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.822 INFO:tasks.workunit.client.0.mira109.stderr:7f3438035000-7f343c000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.822 INFO:tasks.workunit.client.0.mira109.stderr:7f343c000000-7f343c021000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.822 INFO:tasks.workunit.client.0.mira109.stderr:7f343c021000-7f3440000000 ---p 00000000 00:00 0 2019-11-20T21:26:49.823 INFO:tasks.workunit.client.0.mira109.stderr:7f3440209000-7f344020a000 ---p 00000000 00:00 0 2019-11-20T21:26:49.823 INFO:tasks.workunit.client.0.mira109.stderr:7f344020a000-7f3440a0a000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.823 INFO:tasks.workunit.client.0.mira109.stderr:7f3440a0a000-7f3440a0b000 ---p 00000000 00:00 0 2019-11-20T21:26:49.823 INFO:tasks.workunit.client.0.mira109.stderr:7f3440a0b000-7f344120b000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.823 INFO:tasks.workunit.client.0.mira109.stderr:7f344120b000-7f344120c000 ---p 00000000 00:00 0 2019-11-20T21:26:49.823 INFO:tasks.workunit.client.0.mira109.stderr:7f344120c000-7f3441a0c000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.823 INFO:tasks.workunit.client.0.mira109.stderr:7f3441a0c000-7f3441a0d000 ---p 00000000 00:00 0 2019-11-20T21:26:49.824 INFO:tasks.workunit.client.0.mira109.stderr:7f3441a0d000-7f344220d000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.824 INFO:tasks.workunit.client.0.mira109.stderr:7f344220d000-7f344220e000 ---p 00000000 00:00 0 2019-11-20T21:26:49.824 INFO:tasks.workunit.client.0.mira109.stderr:7f344220e000-7f3442a0e000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.824 INFO:tasks.workunit.client.0.mira109.stderr:7f3442a0e000-7f3442a0f000 ---p 00000000 00:00 0 2019-11-20T21:26:49.824 INFO:tasks.workunit.client.0.mira109.stderr:7f3442a0f000-7f344320f000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.824 INFO:tasks.workunit.client.0.mira109.stderr:7f344320f000-7f3443210000 ---p 00000000 00:00 0 2019-11-20T21:26:49.824 INFO:tasks.workunit.client.0.mira109.stderr:7f3443210000-7f3443a10000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.825 INFO:tasks.workunit.client.0.mira109.stderr:7f3443a10000-7f3443a11000 ---p 00000000 00:00 0 2019-11-20T21:26:49.825 INFO:tasks.workunit.client.0.mira109.stderr:7f3443a11000-7f3444211000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.825 INFO:tasks.workunit.client.0.mira109.stderr:7f3444211000-7f3444212000 ---p 00000000 00:00 0 2019-11-20T21:26:49.825 INFO:tasks.workunit.client.0.mira109.stderr:7f3444212000-7f3444a12000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.825 INFO:tasks.workunit.client.0.mira109.stderr:7f3444a12000-7f3444a13000 ---p 00000000 00:00 0 2019-11-20T21:26:49.825 INFO:tasks.workunit.client.0.mira109.stderr:7f3444a13000-7f3445c74000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.825 INFO:tasks.workunit.client.0.mira109.stderr:7f3445c74000-7f3445c76000 r-xp 00000000 08:01 73002 /usr/lib64/liburcu-common.so.1.0.0 2019-11-20T21:26:49.825 INFO:tasks.workunit.client.0.mira109.stderr:7f3445c76000-7f3445e75000 ---p 00002000 08:01 73002 /usr/lib64/liburcu-common.so.1.0.0 2019-11-20T21:26:49.826 INFO:tasks.workunit.client.0.mira109.stderr:7f3445e75000-7f3445e76000 r--p 00001000 08:01 73002 /usr/lib64/liburcu-common.so.1.0.0 2019-11-20T21:26:49.826 INFO:tasks.workunit.client.0.mira109.stderr:7f3445e76000-7f3445e77000 rw-p 00002000 08:01 73002 /usr/lib64/liburcu-common.so.1.0.0 2019-11-20T21:26:49.826 INFO:tasks.workunit.client.0.mira109.stderr:7f3445e77000-7f3445e7d000 r-xp 00000000 08:01 73000 /usr/lib64/liburcu-cds.so.1.0.0 2019-11-20T21:26:49.826 INFO:tasks.workunit.client.0.mira109.stderr:7f3445e7d000-7f344607c000 ---p 00006000 08:01 73000 /usr/lib64/liburcu-cds.so.1.0.0 2019-11-20T21:26:49.826 INFO:tasks.workunit.client.0.mira109.stderr:7f344607c000-7f344607d000 r--p 00005000 08:01 73000 /usr/lib64/liburcu-cds.so.1.0.0 2019-11-20T21:26:49.826 INFO:tasks.workunit.client.0.mira109.stderr:7f344607d000-7f344607e000 rw-p 00006000 08:01 73000 /usr/lib64/liburcu-cds.so.1.0.0 2019-11-20T21:26:49.826 INFO:tasks.workunit.client.0.mira109.stderr:7f344607e000-7f3446084000 r-xp 00000000 08:01 72998 /usr/lib64/liburcu-bp.so.1.0.0 2019-11-20T21:26:49.827 INFO:tasks.workunit.client.0.mira109.stderr:7f3446084000-7f3446284000 ---p 00006000 08:01 72998 /usr/lib64/liburcu-bp.so.1.0.0 2019-11-20T21:26:49.827 INFO:tasks.workunit.client.0.mira109.stderr:7f3446284000-7f3446285000 r--p 00006000 08:01 72998 /usr/lib64/liburcu-bp.so.1.0.0 2019-11-20T21:26:49.827 INFO:tasks.workunit.client.0.mira109.stderr:7f3446285000-7f3446286000 rw-p 00007000 08:01 72998 /usr/lib64/liburcu-bp.so.1.0.0 2019-11-20T21:26:49.827 INFO:tasks.workunit.client.0.mira109.stderr:7f3446286000-7f344628f000 r-xp 00000000 08:01 73163 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2019-11-20T21:26:49.827 INFO:tasks.workunit.client.0.mira109.stderr:7f344628f000-7f344648e000 ---p 00009000 08:01 73163 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2019-11-20T21:26:49.827 INFO:tasks.workunit.client.0.mira109.stderr:7f344648e000-7f344648f000 r--p 00008000 08:01 73163 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2019-11-20T21:26:49.827 INFO:tasks.workunit.client.0.mira109.stderr:7f344648f000-7f3446490000 rw-p 00009000 08:01 73163 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2019-11-20T21:26:49.828 INFO:tasks.workunit.client.0.mira109.stderr:7f3446490000-7f34464a0000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.828 INFO:tasks.workunit.client.0.mira109.stderr:7f34464a0000-7f34464af000 r-xp 00000000 08:01 4212 /usr/lib64/libbz2.so.1.0.6 2019-11-20T21:26:49.828 INFO:tasks.workunit.client.0.mira109.stderr:7f34464af000-7f34466ae000 ---p 0000f000 08:01 4212 /usr/lib64/libbz2.so.1.0.6 2019-11-20T21:26:49.828 INFO:tasks.workunit.client.0.mira109.stderr:7f34466ae000-7f34466af000 r--p 0000e000 08:01 4212 /usr/lib64/libbz2.so.1.0.6 2019-11-20T21:26:49.828 INFO:tasks.workunit.client.0.mira109.stderr:7f34466af000-7f34466b0000 rw-p 0000f000 08:01 4212 /usr/lib64/libbz2.so.1.0.6 2019-11-20T21:26:49.829 INFO:tasks.workunit.client.0.mira109.stderr:7f34466b0000-7f34466d5000 r-xp 00000000 08:01 4203 /usr/lib64/liblzma.so.5.2.2 2019-11-20T21:26:49.829 INFO:tasks.workunit.client.0.mira109.stderr:7f34466d5000-7f34468d4000 ---p 00025000 08:01 4203 /usr/lib64/liblzma.so.5.2.2 2019-11-20T21:26:49.829 INFO:tasks.workunit.client.0.mira109.stderr:7f34468d4000-7f34468d5000 r--p 00024000 08:01 4203 /usr/lib64/liblzma.so.5.2.2 2019-11-20T21:26:49.829 INFO:tasks.workunit.client.0.mira109.stderr:7f34468d5000-7f34468d6000 rw-p 00025000 08:01 4203 /usr/lib64/liblzma.so.5.2.2 2019-11-20T21:26:49.829 INFO:tasks.workunit.client.0.mira109.stderr:7f34468d6000-7f34468ed000 r-xp 00000000 08:01 4213 /usr/lib64/libelf-0.172.so 2019-11-20T21:26:49.829 INFO:tasks.workunit.client.0.mira109.stderr:7f34468ed000-7f3446aec000 ---p 00017000 08:01 4213 /usr/lib64/libelf-0.172.so 2019-11-20T21:26:49.829 INFO:tasks.workunit.client.0.mira109.stderr:7f3446aec000-7f3446aed000 r--p 00016000 08:01 4213 /usr/lib64/libelf-0.172.so 2019-11-20T21:26:49.830 INFO:tasks.workunit.client.0.mira109.stderr:7f3446aed000-7f3446aee000 rw-p 00017000 08:01 4213 /usr/lib64/libelf-0.172.so 2019-11-20T21:26:49.830 INFO:tasks.workunit.client.0.mira109.stderr:7f3446aee000-7f3446af2000 r-xp 00000000 08:01 3809 /usr/lib64/libattr.so.1.1.0 2019-11-20T21:26:49.830 INFO:tasks.workunit.client.0.mira109.stderr:7f3446af2000-7f3446cf1000 ---p 00004000 08:01 3809 /usr/lib64/libattr.so.1.1.0 2019-11-20T21:26:49.830 INFO:tasks.workunit.client.0.mira109.stderr:7f3446cf1000-7f3446cf2000 r--p 00003000 08:01 3809 /usr/lib64/libattr.so.1.1.0 2019-11-20T21:26:49.830 INFO:tasks.workunit.client.0.mira109.stderr:7f3446cf2000-7f3446cf3000 rw-p 00004000 08:01 3809 /usr/lib64/libattr.so.1.1.0 2019-11-20T21:26:49.830 INFO:tasks.workunit.client.0.mira109.stderr:7f3446cf3000-7f3446d11000 r-xp 00000000 08:01 4428 /usr/lib64/libnl-3.so.200.23.0 2019-11-20T21:26:49.830 INFO:tasks.workunit.client.0.mira109.stderr:7f3446d11000-7f3446f11000 ---p 0001e000 08:01 4428 /usr/lib64/libnl-3.so.200.23.0 2019-11-20T21:26:49.831 INFO:tasks.workunit.client.0.mira109.stderr:7f3446f11000-7f3446f13000 r--p 0001e000 08:01 4428 /usr/lib64/libnl-3.so.200.23.0 2019-11-20T21:26:49.831 INFO:tasks.workunit.client.0.mira109.stderr:7f3446f13000-7f3446f14000 rw-p 00020000 08:01 4428 /usr/lib64/libnl-3.so.200.23.0 2019-11-20T21:26:49.831 INFO:tasks.workunit.client.0.mira109.stderr:7f3446f14000-7f3446f78000 r-xp 00000000 08:01 4436 /usr/lib64/libnl-route-3.so.200.23.0 2019-11-20T21:26:49.831 INFO:tasks.workunit.client.0.mira109.stderr:7f3446f78000-7f3447177000 ---p 00064000 08:01 4436 /usr/lib64/libnl-route-3.so.200.23.0 2019-11-20T21:26:49.831 INFO:tasks.workunit.client.0.mira109.stderr:7f3447177000-7f344717a000 r--p 00063000 08:01 4436 /usr/lib64/libnl-route-3.so.200.23.0 2019-11-20T21:26:49.831 INFO:tasks.workunit.client.0.mira109.stderr:7f344717a000-7f344717f000 rw-p 00066000 08:01 4436 /usr/lib64/libnl-route-3.so.200.23.0 2019-11-20T21:26:49.831 INFO:tasks.workunit.client.0.mira109.stderr:7f344717f000-7f3447181000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.832 INFO:tasks.workunit.client.0.mira109.stderr:7f3447181000-7f34471cd000 r-xp 00000000 08:01 7367 /usr/lib64/libdw-0.172.so 2019-11-20T21:26:49.832 INFO:tasks.workunit.client.0.mira109.stderr:7f34471cd000-7f34473cd000 ---p 0004c000 08:01 7367 /usr/lib64/libdw-0.172.so 2019-11-20T21:26:49.832 INFO:tasks.workunit.client.0.mira109.stderr:7f34473cd000-7f34473cf000 r--p 0004c000 08:01 7367 /usr/lib64/libdw-0.172.so 2019-11-20T21:26:49.832 INFO:tasks.workunit.client.0.mira109.stderr:7f34473cf000-7f34473d0000 rw-p 0004e000 08:01 7367 /usr/lib64/libdw-0.172.so 2019-11-20T21:26:49.832 INFO:tasks.workunit.client.0.mira109.stderr:7f34473d0000-7f34473d4000 r-xp 00000000 08:01 3811 /usr/lib64/libcap.so.2.22 2019-11-20T21:26:49.832 INFO:tasks.workunit.client.0.mira109.stderr:7f34473d4000-7f34475d3000 ---p 00004000 08:01 3811 /usr/lib64/libcap.so.2.22 2019-11-20T21:26:49.833 INFO:tasks.workunit.client.0.mira109.stderr:7f34475d3000-7f34475d4000 r--p 00003000 08:01 3811 /usr/lib64/libcap.so.2.22 2019-11-20T21:26:49.833 INFO:tasks.workunit.client.0.mira109.stderr:7f34475d4000-7f34475d5000 rw-p 00004000 08:01 3811 /usr/lib64/libcap.so.2.22 2019-11-20T21:26:49.833 INFO:tasks.workunit.client.0.mira109.stderr:7f34475d5000-7f34475d9000 r-xp 00000000 08:01 4210 /usr/lib64/libuuid.so.1.3.0 2019-11-20T21:26:49.833 INFO:tasks.workunit.client.0.mira109.stderr:7f34475d9000-7f34477d8000 ---p 00004000 08:01 4210 /usr/lib64/libuuid.so.1.3.0 2019-11-20T21:26:49.833 INFO:tasks.workunit.client.0.mira109.stderr:7f34477d8000-7f34477d9000 r--p 00003000 08:01 4210 /usr/lib64/libuuid.so.1.3.0 2019-11-20T21:26:49.833 INFO:tasks.workunit.client.0.mira109.stderr:7f34477d9000-7f34477da000 rw-p 00004000 08:01 4210 /usr/lib64/libuuid.so.1.3.0 2019-11-20T21:26:49.834 INFO:tasks.workunit.client.0.mira109.stderr:7f34477da000-7f344799c000 r-xp 00000000 08:01 3704 /usr/lib64/libc-2.17.so 2019-11-20T21:26:49.834 INFO:tasks.workunit.client.0.mira109.stderr:7f344799c000-7f3447b9c000 ---p 001c2000 08:01 3704 /usr/lib64/libc-2.17.so 2019-11-20T21:26:49.834 INFO:tasks.workunit.client.0.mira109.stderr:7f3447b9c000-7f3447ba0000 r--p 001c2000 08:01 3704 /usr/lib64/libc-2.17.so 2019-11-20T21:26:49.834 INFO:tasks.workunit.client.0.mira109.stderr:7f3447ba0000-7f3447ba2000 rw-p 001c6000 08:01 3704 /usr/lib64/libc-2.17.so 2019-11-20T21:26:49.834 INFO:tasks.workunit.client.0.mira109.stderr:7f3447ba2000-7f3447ba7000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.834 INFO:tasks.workunit.client.0.mira109.stderr:7f3447ba7000-7f3447bbc000 r-xp 00000000 08:01 12863 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 2019-11-20T21:26:49.834 INFO:tasks.workunit.client.0.mira109.stderr:7f3447bbc000-7f3447dbb000 ---p 00015000 08:01 12863 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 2019-11-20T21:26:49.835 INFO:tasks.workunit.client.0.mira109.stderr:7f3447dbb000-7f3447dbc000 r--p 00014000 08:01 12863 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 2019-11-20T21:26:49.835 INFO:tasks.workunit.client.0.mira109.stderr:7f3447dbc000-7f3447dbd000 rw-p 00015000 08:01 12863 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 2019-11-20T21:26:49.835 INFO:tasks.workunit.client.0.mira109.stderr:7f3447dbd000-7f3447ebe000 r-xp 00000000 08:01 3709 /usr/lib64/libm-2.17.so 2019-11-20T21:26:49.835 INFO:tasks.workunit.client.0.mira109.stderr:7f3447ebe000-7f34480bd000 ---p 00101000 08:01 3709 /usr/lib64/libm-2.17.so 2019-11-20T21:26:49.835 INFO:tasks.workunit.client.0.mira109.stderr:7f34480bd000-7f34480be000 r--p 00100000 08:01 3709 /usr/lib64/libm-2.17.so 2019-11-20T21:26:49.835 INFO:tasks.workunit.client.0.mira109.stderr:7f34480be000-7f34480bf000 rw-p 00101000 08:01 3709 /usr/lib64/libm-2.17.so 2019-11-20T21:26:49.835 INFO:tasks.workunit.client.0.mira109.stderr:7f34480bf000-7f34481a8000 r-xp 00000000 08:01 3738 /usr/lib64/libstdc++.so.6.0.19 2019-11-20T21:26:49.836 INFO:tasks.workunit.client.0.mira109.stderr:7f34481a8000-7f34483a7000 ---p 000e9000 08:01 3738 /usr/lib64/libstdc++.so.6.0.19 2019-11-20T21:26:49.836 INFO:tasks.workunit.client.0.mira109.stderr:7f34483a7000-7f34483af000 r--p 000e8000 08:01 3738 /usr/lib64/libstdc++.so.6.0.19 2019-11-20T21:26:49.836 INFO:tasks.workunit.client.0.mira109.stderr:7f34483af000-7f34483b1000 rw-p 000f0000 08:01 3738 /usr/lib64/libstdc++.so.6.0.19 2019-11-20T21:26:49.836 INFO:tasks.workunit.client.0.mira109.stderr:7f34483b1000-7f34483c6000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.836 INFO:tasks.workunit.client.0.mira109.stderr:7f34483c6000-7f34483c9000 r-xp 00000000 08:01 3955 /usr/lib64/libkeyutils.so.1.5 2019-11-20T21:26:49.836 INFO:tasks.workunit.client.0.mira109.stderr:7f34483c9000-7f34485c8000 ---p 00003000 08:01 3955 /usr/lib64/libkeyutils.so.1.5 2019-11-20T21:26:49.837 INFO:tasks.workunit.client.0.mira109.stderr:7f34485c8000-7f34485c9000 r--p 00002000 08:01 3955 /usr/lib64/libkeyutils.so.1.5 2019-11-20T21:26:49.837 INFO:tasks.workunit.client.0.mira109.stderr:7f34485c9000-7f34485ca000 rw-p 00003000 08:01 3955 /usr/lib64/libkeyutils.so.1.5 2019-11-20T21:26:49.837 INFO:tasks.workunit.client.0.mira109.stderr:7f34485ca000-7f34485e0000 r-xp 00000000 08:01 3724 /usr/lib64/libresolv-2.17.so 2019-11-20T21:26:49.837 INFO:tasks.workunit.client.0.mira109.stderr:7f34485e0000-7f34487df000 ---p 00016000 08:01 3724 /usr/lib64/libresolv-2.17.so 2019-11-20T21:26:49.837 INFO:tasks.workunit.client.0.mira109.stderr:7f34487df000-7f34487e0000 r--p 00015000 08:01 3724 /usr/lib64/libresolv-2.17.so 2019-11-20T21:26:49.837 INFO:tasks.workunit.client.0.mira109.stderr:7f34487e0000-7f34487e1000 rw-p 00016000 08:01 3724 /usr/lib64/libresolv-2.17.so 2019-11-20T21:26:49.837 INFO:tasks.workunit.client.0.mira109.stderr:7f34487e1000-7f34487e3000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.838 INFO:tasks.workunit.client.0.mira109.stderr:7f34487e3000-7f34487ea000 r-xp 00000000 08:01 3725 /usr/lib64/librt-2.17.so 2019-11-20T21:26:49.838 INFO:tasks.workunit.client.0.mira109.stderr:7f34487ea000-7f34489e9000 ---p 00007000 08:01 3725 /usr/lib64/librt-2.17.so 2019-11-20T21:26:49.838 INFO:tasks.workunit.client.0.mira109.stderr:7f34489e9000-7f34489ea000 r--p 00006000 08:01 3725 /usr/lib64/librt-2.17.so 2019-11-20T21:26:49.838 INFO:tasks.workunit.client.0.mira109.stderr:7f34489ea000-7f34489eb000 rw-p 00007000 08:01 3725 /usr/lib64/librt-2.17.so 2019-11-20T21:26:49.838 INFO:tasks.workunit.client.0.mira109.stderr:7f34489eb000-7f3448a00000 r-xp 00000000 08:01 3737 /usr/lib64/libz.so.1.2.7 2019-11-20T21:26:49.838 INFO:tasks.workunit.client.0.mira109.stderr:7f3448a00000-7f3448bff000 ---p 00015000 08:01 3737 /usr/lib64/libz.so.1.2.7 2019-11-20T21:26:49.838 INFO:tasks.workunit.client.0.mira109.stderr:7f3448bff000-7f3448c00000 r--p 00014000 08:01 3737 /usr/lib64/libz.so.1.2.7 2019-11-20T21:26:49.838 INFO:tasks.workunit.client.0.mira109.stderr:7f3448c00000-7f3448c01000 rw-p 00015000 08:01 3737 /usr/lib64/libz.so.1.2.7 2019-11-20T21:26:49.840 INFO:tasks.workunit.client.0.mira109.stderr:7f3448c01000-7f3448c16000 r-xp 00000000 08:01 13444 /usr/lib64/librdmacm.so.1.1.17.2 2019-11-20T21:26:49.840 INFO:tasks.workunit.client.0.mira109.stderr:7f3448c16000-7f3448e15000 ---p 00015000 08:01 13444 /usr/lib64/librdmacm.so.1.1.17.2 2019-11-20T21:26:49.840 INFO:tasks.workunit.client.0.mira109.stderr:7f3448e15000-7f3448e16000 r--p 00014000 08:01 13444 /usr/lib64/librdmacm.so.1.1.17.2 2019-11-20T21:26:49.840 INFO:tasks.workunit.client.0.mira109.stderr:7f3448e16000-7f3448e17000 rw-p 00015000 08:01 13444 /usr/lib64/librdmacm.so.1.1.17.2 2019-11-20T21:26:49.840 INFO:tasks.workunit.client.0.mira109.stderr:7f3448e17000-7f3448e18000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.840 INFO:tasks.workunit.client.0.mira109.stderr:7f3448e18000-7f3448e2d000 r-xp 00000000 08:01 13434 /usr/lib64/libibverbs.so.1.1.17.2 2019-11-20T21:26:49.840 INFO:tasks.workunit.client.0.mira109.stderr:7f3448e2d000-7f344902d000 ---p 00015000 08:01 13434 /usr/lib64/libibverbs.so.1.1.17.2 2019-11-20T21:26:49.841 INFO:tasks.workunit.client.0.mira109.stderr:7f344902d000-7f344902e000 r--p 00015000 08:01 13434 /usr/lib64/libibverbs.so.1.1.17.2 2019-11-20T21:26:49.841 INFO:tasks.workunit.client.0.mira109.stderr:7f344902e000-7f344902f000 rw-p 00016000 08:01 13434 /usr/lib64/libibverbs.so.1.1.17.2 2019-11-20T21:26:49.841 INFO:tasks.workunit.client.0.mira109.stderr:7f344902f000-7f3449044000 r-xp 00000000 08:01 7382 /usr/lib64/libudev.so.1.6.2 2019-11-20T21:26:49.841 INFO:tasks.workunit.client.0.mira109.stderr:7f3449044000-7f3449243000 ---p 00015000 08:01 7382 /usr/lib64/libudev.so.1.6.2 2019-11-20T21:26:49.841 INFO:tasks.workunit.client.0.mira109.stderr:7f3449243000-7f3449244000 r--p 00014000 08:01 7382 /usr/lib64/libudev.so.1.6.2 2019-11-20T21:26:49.841 INFO:tasks.workunit.client.0.mira109.stderr:7f3449244000-7f3449245000 rw-p 00015000 08:01 7382 /usr/lib64/libudev.so.1.6.2 2019-11-20T21:26:49.841 INFO:tasks.workunit.client.0.mira109.stderr:7f3449245000-7f344925c000 r-xp 00000000 08:01 3723 /usr/lib64/libpthread-2.17.so 2019-11-20T21:26:49.842 INFO:tasks.workunit.client.0.mira109.stderr:7f344925c000-7f344945b000 ---p 00017000 08:01 3723 /usr/lib64/libpthread-2.17.so 2019-11-20T21:26:49.842 INFO:tasks.workunit.client.0.mira109.stderr:7f344945b000-7f344945c000 r--p 00016000 08:01 3723 /usr/lib64/libpthread-2.17.so 2019-11-20T21:26:49.842 INFO:tasks.workunit.client.0.mira109.stderr:7f344945c000-7f344945d000 rw-p 00017000 08:01 3723 /usr/lib64/libpthread-2.17.so 2019-11-20T21:26:49.842 INFO:tasks.workunit.client.0.mira109.stderr:7f344945d000-7f3449461000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.842 INFO:tasks.workunit.client.0.mira109.stderr:7f3449461000-7f3449695000 r-xp 00000000 08:01 3992 /usr/lib64/libcrypto.so.1.0.2k 2019-11-20T21:26:49.842 INFO:tasks.workunit.client.0.mira109.stderr:7f3449695000-7f3449895000 ---p 00234000 08:01 3992 /usr/lib64/libcrypto.so.1.0.2k 2019-11-20T21:26:49.842 INFO:tasks.workunit.client.0.mira109.stderr:7f3449895000-7f34498b1000 r--p 00234000 08:01 3992 /usr/lib64/libcrypto.so.1.0.2k 2019-11-20T21:26:49.843 INFO:tasks.workunit.client.0.mira109.stderr:7f34498b1000-7f34498be000 rw-p 00250000 08:01 3992 /usr/lib64/libcrypto.so.1.0.2k 2019-11-20T21:26:49.843 INFO:tasks.workunit.client.0.mira109.stderr:7f34498be000-7f34498c2000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.843 INFO:tasks.workunit.client.0.mira109.stderr:7f34498c2000-7f344a293000 r-xp 00000000 08:01 395625 /usr/lib64/ceph/libceph-common.so.2 2019-11-20T21:26:49.843 INFO:tasks.workunit.client.0.mira109.stderr:7f344a293000-7f344a493000 ---p 009d1000 08:01 395625 /usr/lib64/ceph/libceph-common.so.2 2019-11-20T21:26:49.843 INFO:tasks.workunit.client.0.mira109.stderr:7f344a493000-7f344a4b3000 r--p 009d1000 08:01 395625 /usr/lib64/ceph/libceph-common.so.2 2019-11-20T21:26:49.843 INFO:tasks.workunit.client.0.mira109.stderr:7f344a4b3000-7f344a4bc000 rw-p 009f1000 08:01 395625 /usr/lib64/ceph/libceph-common.so.2 2019-11-20T21:26:49.843 INFO:tasks.workunit.client.0.mira109.stderr:7f344a4bc000-7f34526dc000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.844 INFO:tasks.workunit.client.0.mira109.stderr:7f34526dc000-7f34526de000 r-xp 00000000 08:01 3708 /usr/lib64/libdl-2.17.so 2019-11-20T21:26:49.844 INFO:tasks.workunit.client.0.mira109.stderr:7f34526de000-7f34528de000 ---p 00002000 08:01 3708 /usr/lib64/libdl-2.17.so 2019-11-20T21:26:49.844 INFO:tasks.workunit.client.0.mira109.stderr:7f34528de000-7f34528df000 r--p 00002000 08:01 3708 /usr/lib64/libdl-2.17.so 2019-11-20T21:26:49.845 INFO:tasks.workunit.client.0.mira109.stderr:7f34528df000-7f34528e0000 rw-p 00003000 08:01 3708 /usr/lib64/libdl-2.17.so 2019-11-20T21:26:49.845 INFO:tasks.workunit.client.0.mira109.stderr:7f34528e0000-7f345291c000 r-xp 00000000 08:01 4222 /usr/lib64/libblkid.so.1.1.0 2019-11-20T21:26:49.845 INFO:tasks.workunit.client.0.mira109.stderr:7f345291c000-7f3452b1b000 ---p 0003c000 08:01 4222 /usr/lib64/libblkid.so.1.1.0 2019-11-20T21:26:49.845 INFO:tasks.workunit.client.0.mira109.stderr:7f3452b1b000-7f3452b1e000 r--p 0003b000 08:01 4222 /usr/lib64/libblkid.so.1.1.0 2019-11-20T21:26:49.845 INFO:tasks.workunit.client.0.mira109.stderr:7f3452b1e000-7f3452b1f000 rw-p 0003e000 08:01 4222 /usr/lib64/libblkid.so.1.1.0 2019-11-20T21:26:49.845 INFO:tasks.workunit.client.0.mira109.stderr:7f3452b1f000-7f3452b20000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.846 INFO:tasks.workunit.client.0.mira109.stderr:7f3452b20000-7f3452b2e000 r-xp 00000000 08:01 3741 /usr/lib64/libform.so.5.9 2019-11-20T21:26:49.846 INFO:tasks.workunit.client.0.mira109.stderr:7f3452b2e000-7f3452d2e000 ---p 0000e000 08:01 3741 /usr/lib64/libform.so.5.9 2019-11-20T21:26:49.846 INFO:tasks.workunit.client.0.mira109.stderr:7f3452d2e000-7f3452d2f000 r--p 0000e000 08:01 3741 /usr/lib64/libform.so.5.9 2019-11-20T21:26:49.846 INFO:tasks.workunit.client.0.mira109.stderr:7f3452d2f000-7f3452d30000 rw-p 0000f000 08:01 3741 /usr/lib64/libform.so.5.9 2019-11-20T21:26:49.846 INFO:tasks.workunit.client.0.mira109.stderr:7f3452d30000-7f3452d55000 r-xp 00000000 08:01 3763 /usr/lib64/libtinfo.so.5.9 2019-11-20T21:26:49.846 INFO:tasks.workunit.client.0.mira109.stderr:7f3452d55000-7f3452f55000 ---p 00025000 08:01 3763 /usr/lib64/libtinfo.so.5.9 2019-11-20T21:26:49.847 INFO:tasks.workunit.client.0.mira109.stderr:7f3452f55000-7f3452f59000 r--p 00025000 08:01 3763 /usr/lib64/libtinfo.so.5.9 2019-11-20T21:26:49.847 INFO:tasks.workunit.client.0.mira109.stderr:7f3452f59000-7f3452f5a000 rw-p 00029000 08:01 3763 /usr/lib64/libtinfo.so.5.9 2019-11-20T21:26:49.847 INFO:tasks.workunit.client.0.mira109.stderr:7f3452f5a000-7f3452f80000 r-xp 00000000 08:01 3753 /usr/lib64/libncurses.so.5.9 2019-11-20T21:26:49.847 INFO:tasks.workunit.client.0.mira109.stderr:7f3452f80000-7f345317f000 ---p 00026000 08:01 3753 /usr/lib64/libncurses.so.5.9 2019-11-20T21:26:49.847 INFO:tasks.workunit.client.0.mira109.stderr:7f345317f000-7f3453180000 r--p 00025000 08:01 3753 /usr/lib64/libncurses.so.5.9 2019-11-20T21:26:49.847 INFO:tasks.workunit.client.0.mira109.stderr:7f3453180000-7f3453181000 rw-p 00026000 08:01 3753 /usr/lib64/libncurses.so.5.9 2019-11-20T21:26:49.847 INFO:tasks.workunit.client.0.mira109.stderr:7f3453181000-7f34532f0000 r-xp 00000000 08:01 13419 /usr/lib64/librados.so.2.0.0 2019-11-20T21:26:49.848 INFO:tasks.workunit.client.0.mira109.stderr:7f34532f0000-7f34534ef000 ---p 0016f000 08:01 13419 /usr/lib64/librados.so.2.0.0 2019-11-20T21:26:49.848 INFO:tasks.workunit.client.0.mira109.stderr:7f34534ef000-7f34534f6000 r--p 0016e000 08:01 13419 /usr/lib64/librados.so.2.0.0 2019-11-20T21:26:49.848 INFO:tasks.workunit.client.0.mira109.stderr:7f34534f6000-7f34534ff000 rw-p 00175000 08:01 13419 /usr/lib64/librados.so.2.0.0 2019-11-20T21:26:49.848 INFO:tasks.workunit.client.0.mira109.stderr:7f34534ff000-7f34539e8000 r-xp 00000000 08:01 13446 /usr/lib64/librbd.so.1.12.0 2019-11-20T21:26:49.848 INFO:tasks.workunit.client.0.mira109.stderr:7f34539e8000-7f3453be8000 ---p 004e9000 08:01 13446 /usr/lib64/librbd.so.1.12.0 2019-11-20T21:26:49.848 INFO:tasks.workunit.client.0.mira109.stderr:7f3453be8000-7f3453bff000 r--p 004e9000 08:01 13446 /usr/lib64/librbd.so.1.12.0 2019-11-20T21:26:49.848 INFO:tasks.workunit.client.0.mira109.stderr:7f3453bff000-7f3453c05000 rw-p 00500000 08:01 13446 /usr/lib64/librbd.so.1.12.0 2019-11-20T21:26:49.849 INFO:tasks.workunit.client.0.mira109.stderr:7f3453c05000-7f3453c07000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.849 INFO:tasks.workunit.client.0.mira109.stderr:7f3453c07000-7f3453c29000 r-xp 00000000 08:01 3494 /usr/lib64/ld-2.17.so 2019-11-20T21:26:49.849 INFO:tasks.workunit.client.0.mira109.stderr:7f3453ceb000-7f3453d9c000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.849 INFO:tasks.workunit.client.0.mira109.stderr:7f3453df5000-7f3453e1a000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.849 INFO:tasks.workunit.client.0.mira109.stderr:7f3453e26000-7f3453e28000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.849 INFO:tasks.workunit.client.0.mira109.stderr:7f3453e28000-7f3453e29000 r--p 00021000 08:01 3494 /usr/lib64/ld-2.17.so 2019-11-20T21:26:49.849 INFO:tasks.workunit.client.0.mira109.stderr:7f3453e29000-7f3453e2a000 rw-p 00022000 08:01 3494 /usr/lib64/ld-2.17.so 2019-11-20T21:26:49.850 INFO:tasks.workunit.client.0.mira109.stderr:7f3453e2a000-7f3453e2b000 rw-p 00000000 00:00 0 2019-11-20T21:26:49.850 INFO:tasks.workunit.client.0.mira109.stderr:7ffc72088000-7ffc72142000 rw-p 00000000 00:00 0 [stack] 2019-11-20T21:26:49.850 INFO:tasks.workunit.client.0.mira109.stderr:7ffc721eb000-7ffc721ee000 r--p 00000000 00:00 0 [vvar] 2019-11-20T21:26:49.850 INFO:tasks.workunit.client.0.mira109.stderr:7ffc721ee000-7ffc721ef000 r-xp 00000000 00:00 0 [vdso] 2019-11-20T21:26:49.850 INFO:tasks.workunit.client.0.mira109.stderr:ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] 2019-11-20T21:26:49.850 INFO:tasks.workunit.client.0.mira109.stderr:*** Caught signal (Aborted) ** 2019-11-20T21:26:49.850 INFO:tasks.workunit.client.0.mira109.stderr: in thread 7f342a7fc700 thread_name:tp_librbd 2019-11-20T21:26:49.902 INFO:tasks.workunit.client.0.mira109.stderr: ceph version 15.0.0-7426-g15b37e0 (15b37e0f4e7f6930b0d185998c098a2a36d5a6d7) octopus (dev) 2019-11-20T21:26:49.903 INFO:tasks.workunit.client.0.mira109.stderr: 1: (()+0xf5d0) [0x7f34492545d0] 2019-11-20T21:26:49.903 INFO:tasks.workunit.client.0.mira109.stderr: 2: (gsignal()+0x37) [0x7f34478102c7] 2019-11-20T21:26:49.903 INFO:tasks.workunit.client.0.mira109.stderr: 3: (abort()+0x148) [0x7f34478119b8] 2019-11-20T21:26:49.903 INFO:tasks.workunit.client.0.mira109.stderr: 4: (()+0x78e17) [0x7f3447852e17] 2019-11-20T21:26:49.903 INFO:tasks.workunit.client.0.mira109.stderr: 5: (()+0x82e16) [0x7f344785ce16] 2019-11-20T21:26:49.903 INFO:tasks.workunit.client.0.mira109.stderr: 6: (__libc_malloc()+0x4c) [0x7f344785f32c] 2019-11-20T21:26:49.904 INFO:tasks.workunit.client.0.mira109.stderr: 7: (operator new(unsigned long)+0x1d) [0x7f344811decd] 2019-11-20T21:26:49.904 INFO:tasks.workunit.client.0.mira109.stderr: 8: (()+0x20b497) [0x7f345370a497] 2019-11-20T21:26:49.904 INFO:tasks.workunit.client.0.mira109.stderr: 9: (()+0x20b867) [0x7f345370a867] 2019-11-20T21:26:49.904 INFO:tasks.workunit.client.0.mira109.stderr: 10: (()+0xcf969) [0x7f34535ce969] 2019-11-20T21:26:49.904 INFO:tasks.workunit.client.0.mira109.stderr: 11: (ThreadPool::worker(ThreadPool::WorkThread*)+0xd70) [0x7f3449c29ff0] 2019-11-20T21:26:49.904 INFO:tasks.workunit.client.0.mira109.stderr: 12: (ThreadPool::WorkThread::entry()+0x11) [0x7f3449c2a801] 2019-11-20T21:26:49.904 INFO:tasks.workunit.client.0.mira109.stderr: 13: (()+0x7dd5) [0x7f344924cdd5] 2019-11-20T21:26:49.905 INFO:tasks.workunit.client.0.mira109.stderr: 14: (clone()+0x6d) [0x7f34478d802d] 2019-11-20T21:26:49.905 INFO:tasks.workunit.client.0.mira109.stderr:2019-11-20T21:26:49.810+0000 7f342a7fc700 -1 *** Caught signal (Aborted) ** 2019-11-20T21:26:49.905 INFO:tasks.workunit.client.0.mira109.stderr: in thread 7f342a7fc700 thread_name:tp_librbd
Files
Updated by Ilya Dryomov over 4 years ago
The stack is corrupt in both:
(gdb) bt
#0 0x00007f9ae922c49b in raise () from /lib64/libpthread.so.0
#1 0x0000563808fb90a2 in reraise_fatal (signum=11)
at /usr/src/debug/ceph-15.0.0-7426-g15b37e0/src/global/signal_handler.cc:326
#2 handle_fatal_signal(int) ()
at /usr/src/debug/ceph-15.0.0-7426-g15b37e0/src/global/signal_handler.cc:326
#3 0x00007f9ae922c5d0 in _L_unlock_13 () from /lib64/libpthread.so.0
#4 0x00007f9ae922c5bf in _L_unlock_13 () from /lib64/libpthread.so.0
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) bt
#0 0x00007f344925449b in raise () from /lib64/libpthread.so.0
#1 0x000055b8612690a2 in reraise_fatal (signum=6)
at /usr/src/debug/ceph-15.0.0-7426-g15b37e0/src/global/signal_handler.cc:326
#2 handle_fatal_signal(int) ()
at /usr/src/debug/ceph-15.0.0-7426-g15b37e0/src/global/signal_handler.cc:326
#3 0x00007f34492545d0 in _L_unlock_13 () from /lib64/libpthread.so.0
#4 0x00007f34492545bf in _L_unlock_13 () from /lib64/libpthread.so.0
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
Updated by Ilya Dryomov over 4 years ago
- Subject changed from memory corruption in master (15b37e0, centos7) to memory corruption and lockups with I-Object
- Status changed from New to 12
Looks real and seems to be introduced with I-Object: no issues with 36f5fcbb97eb ("Merge PR #31672 into master") and 9/10 failures with the following 669453138d89 ("Merge pull request #16715 from adamemerson/wip-I-Object!").
Note that it doesn't always crash, sometimes it locks up.
Updated by Ilya Dryomov over 4 years ago
Got actionable stack traces on 669453138d89:
Core was generated by `rbd snap create testimg1 --snap=snap1'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f2435ffb700 (LWP 9976))]
(gdb) bt
#0 raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x000055ef41101e50 in reraise_fatal (signum=11) at ./src/global/signal_handler.cc:81
#2 handle_fatal_signal (signum=11) at ./src/global/signal_handler.cc:326
#3 <signal handler called>
#4 0x00007f244ae774f1 in fu2::abi_310::detail::type_erasure::invocation_table::function_trait<void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)>::internal_invoker<fu2::abi_310::detail::type_erasure::box<false, librados::IoCtxImpl::aio_watch(object_t const&, librados::AioCompletionImpl*, unsigned long*, librados::v14_2_0::WatchCtx*, librados::v14_2_0::WatchCtx2*, unsigned int, bool)::{lambda(boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)#1}, std::allocator<{lambda(boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)#1}> >, true>::invoke(fu2::abi_310::detail::type_erasure::data_accessor*, unsigned long, boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&) () at ./src/log/Entry.h:62
#5 0x00007f244aea72a0 in fu2::abi_310::detail::type_erasure::tables::vtable<fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >::invoke<0ul, fu2::abi_310::detail::type_erasure::data_accessor*, unsigned long const&, boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list>(fu2::abi_310::detail::type_erasure::data_accessor*&&, unsigned long const&, boost::system::error_code&&, unsigned long&&, unsigned long&&, unsigned long&&, ceph::buffer::v14_2_0::list&&) const (this=0x7f2418002a30) at ./src/include/function2.hpp:921
#6 fu2::abi_310::detail::type_erasure::erasure<true, fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >::invoke<0ul, fu2::abi_310::detail::type_erasure::erasure<true, fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >&, boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list>(fu2::abi_310::detail::type_erasure::erasure<true, fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >&, boost::system::error_code&&, unsigned long&&, unsigned long&&, unsigned long&&, ceph::buffer::v14_2_0::list&&) (erasure=...) at ./src/include/function2.hpp:1125
#7 fu2::abi_310::detail::type_erasure::invocation_table::operator_impl<0ul, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)>::operator()(boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&) (args#4=..., args#3=4443, args#2=139792998212528, args#1=81604378630, args#0=..., this=0x7f2418002a20) at ./src/include/function2.hpp:702
#8 Objecter::_do_watch_notify (this=<optimized out>, info=0x7f24180023b0, m=0x7f242800bd80) at ./src/osdc/Objecter.cc:967
#9 0x00007f244aeeb683 in CB_DoWatchNotify::operator() (this=<synthetic pointer>) at ./src/osdc/Objecter.cc:899
#10 boost::asio::asio_handler_invoke<CB_DoWatchNotify> (function=<synthetic pointer>...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/handler_invoke_hook.hpp:69
#11 boost_asio_handler_invoke_helpers::invoke<CB_DoWatchNotify, CB_DoWatchNotify> (context=<synthetic pointer>..., function=<synthetic pointer>...)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#12 boost::asio::system_executor::dispatch<CB_DoWatchNotify, std::allocator<void> > (this=<optimized out>, f=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/impl/system_executor.hpp:39
#13 boost::asio::detail::work_dispatcher<CB_DoWatchNotify>::operator() (this=0x7f2435ff6480) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/work_dispatcher.hpp:58
#14 boost::asio::asio_handler_invoke<boost::asio::detail::work_dispatcher<CB_DoWatchNotify> > (function=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/handler_invoke_hook.hpp:69
#15 boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::work_dispatcher<CB_DoWatchNotify>, boost::asio::detail::work_dispatcher<CB_DoWatchNotify> > (context=..., function=...)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#16 boost::asio::detail::handler_work<boost::asio::detail::work_dispatcher<CB_DoWatchNotify>, boost::asio::system_executor>::complete<boost::asio::detail::work_dispatcher<CB_DoWatchNotify> > (this=<synthetic pointer>,
handler=..., function=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/handler_work.hpp:82
#17 boost::asio::detail::completion_handler<boost::asio::detail::work_dispatcher<CB_DoWatchNotify> >::do_complete (owner=0x55ef41bfc3b0, base=0x7f2420002520)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/completion_handler.hpp:70
#18 0x00007f244ae91535 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=0x55ef41bfc3b0, this=<optimized out>)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/scheduler_operation.hpp:40
#19 boost::asio::detail::strand_service::do_complete (owner=0x55ef41bfc3b0, base=0x55ef41c7d280, ec=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/impl/strand_service.ipp:168
#20 0x00007f244ae93c9f in boost::asio::detail::scheduler_operation::complete (bytes_transferred=<optimized out>, ec=..., owner=0x55ef41bfc3b0, this=<optimized out>)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/scheduler_operation.hpp:40
#21 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x55ef41bfc3b0) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/impl/scheduler.ipp:401
#22 boost::asio::detail::scheduler::run (this=0x55ef41bfc3b0, ec=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/impl/scheduler.ipp:154
#23 0x00007f244ae93e49 in boost::asio::io_context::run (this=0x55ef41c7b290) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/impl/io_context.ipp:62
#24 ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const (__closure=<optimized out>) at ./src/common/async/context_pool.h:66
#25 std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#26 std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_result&&, (ceph::async::io_context_pool::start(short)::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#27 std::thread::_Invoker<std::tuple<ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/include/c++/7/thread:234
#28 std::thread::_Invoker<std::tuple<ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() (this=<optimized out>) at /usr/include/c++/7/thread:243
#29 std::thread::_State_impl<std::thread::_Invoker<std::tuple<ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() (this=<optimized out>) at /usr/include/c++/7/thread:186
#30 0x00007f2440bbb66f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#31 0x00007ffee0aff510 in ?? ()
#32 0x84af4eb8117f8fae in ?? ()
#33 0x84afa667f1bb8fae in ?? ()
#34 0x0000000000000000 in ?? ()
Core was generated by `rbd snap create testimg1 --snap=snap1'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fc3e37fe700 (LWP 9993))]
(gdb) bt
#0 raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x0000564e27a1de50 in reraise_fatal (signum=11) at ./src/global/signal_handler.cc:81
#2 handle_fatal_signal (signum=11) at ./src/global/signal_handler.cc:326
#3 <signal handler called>
#4 0x00007fc3f7c63609 in fu2::abi_310::detail::type_erasure::invocation_table::function_trait<void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)>::internal_invoker<fu2::abi_310::detail::type_erasure::box<false, librados::IoCtxImpl::aio_watch(object_t const&, librados::AioCompletionImpl*, unsigned long*, librados::v14_2_0::WatchCtx*, librados::v14_2_0::WatchCtx2*, unsigned int, bool)::{lambda(boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)#1}, std::allocator<{lambda(boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)#1}> >, true>::invoke(fu2::abi_310::detail::type_erasure::data_accessor*, unsigned long, boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&) () at ./src/log/Entry.h:62
#5 0x00007fc3f7cbc537 in fu2::abi_310::detail::type_erasure::tables::vtable<fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >::invoke<0ul, fu2::abi_310::detail::type_erasure::data_accessor*, unsigned long const&, boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list>(fu2::abi_310::detail::type_erasure::data_accessor*&&, unsigned long const&, boost::system::error_code&&, unsigned long&&, unsigned long&&, unsigned long&&, ceph::buffer::v14_2_0::list&&) const (this=0x7fc3c0002a30) at ./src/include/function2.hpp:921
#6 fu2::abi_310::detail::type_erasure::erasure<true, fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >::invoke<0ul, fu2::abi_310::detail::type_erasure::erasure<true, fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >&, boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list>(fu2::abi_310::detail::type_erasure::erasure<true, fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >&, boost::system::error_code&&, unsigned long&&, unsigned long&&, unsigned long&&, ceph::buffer::v14_2_0::list&&) (erasure=...) at ./src/include/function2.hpp:1125
#7 fu2::abi_310::detail::type_erasure::invocation_table::operator_impl<0ul, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)> >, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&)>::operator()(boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&&) (args#4=..., args#3=0, args#2=<optimized out>, args#1=0, args#0=..., this=0x7fc3c0002a20) at ./src/include/function2.hpp:702
#8 CB_DoWatchError::operator() (this=<synthetic pointer>) at ./src/osdc/Objecter.cc:637
#9 boost::asio::asio_handler_invoke<CB_DoWatchError> (function=<synthetic pointer>...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/handler_invoke_hook.hpp:69
#10 boost_asio_handler_invoke_helpers::invoke<CB_DoWatchError, CB_DoWatchError> (context=<synthetic pointer>..., function=<synthetic pointer>...)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#11 boost::asio::system_executor::dispatch<CB_DoWatchError, std::allocator<void> > (this=<optimized out>, f=<synthetic pointer>) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/impl/system_executor.hpp:39
#12 boost::asio::detail::work_dispatcher<CB_DoWatchError>::operator() (this=<synthetic pointer>) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/work_dispatcher.hpp:58
#13 boost::asio::asio_handler_invoke<boost::asio::detail::work_dispatcher<CB_DoWatchError> > (function=<synthetic pointer>...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/handler_invoke_hook.hpp:69
#14 boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::work_dispatcher<CB_DoWatchError>, boost::asio::detail::work_dispatcher<CB_DoWatchError> > (context=<synthetic pointer>..., function=<synthetic pointer>...)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#15 boost::asio::detail::handler_work<boost::asio::detail::work_dispatcher<CB_DoWatchError>, boost::asio::system_executor>::complete<boost::asio::detail::work_dispatcher<CB_DoWatchError> > (this=<synthetic pointer>,
handler=<synthetic pointer>..., function=<synthetic pointer>...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/handler_work.hpp:82
#16 boost::asio::detail::completion_handler<boost::asio::detail::work_dispatcher<CB_DoWatchError> >::do_complete (owner=<optimized out>, base=0x7fc3c0003660)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/completion_handler.hpp:70
#17 0x00007fc3f7c7d535 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=0x564e296783b0, this=<optimized out>)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/scheduler_operation.hpp:40
#18 boost::asio::detail::strand_service::do_complete (owner=0x564e296783b0, base=0x564e296f94d0, ec=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/impl/strand_service.ipp:168
#19 0x00007fc3f7c7fc9f in boost::asio::detail::scheduler_operation::complete (bytes_transferred=<optimized out>, ec=..., owner=0x564e296783b0, this=<optimized out>)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/scheduler_operation.hpp:40
#20 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x564e296783b0) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/impl/scheduler.ipp:401
#21 boost::asio::detail::scheduler::run (this=0x564e296783b0, ec=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/impl/scheduler.ipp:154
#22 0x00007fc3f7c7fe49 in boost::asio::io_context::run (this=0x564e296f73f0) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/impl/io_context.ipp:62
#23 ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const (__closure=<optimized out>) at ./src/common/async/context_pool.h:66
#24 std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#25 std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_result&&, (ceph::async::io_context_pool::start(short)::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#26 std::thread::_Invoker<std::tuple<ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/include/c++/7/thread:234
#27 std::thread::_Invoker<std::tuple<ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() (this=<optimized out>) at /usr/include/c++/7/thread:243
#28 std::thread::_State_impl<std::thread::_Invoker<std::tuple<ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() (this=<optimized out>) at /usr/include/c++/7/thread:186
#29 0x00007fc3ed9a766f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#30 0x00007ffcc2126f10 in ?? ()
#31 0x1892be504925f613 in ?? ()
#32 0x1892a471c9e1f613 in ?? ()
#33 0x0000000000000000 in ?? ()
Core was generated by `rbd rm testimg1'. Program terminated with signal SIGSEGV, Segmentation fault. #0 raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. [Current thread is 1 (Thread 0x7fbd66ffd700 (LWP 10047))] (gdb) bt #0 raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x000055ddbcfa4e50 in reraise_fatal (signum=11) at ./src/global/signal_handler.cc:81 #2 handle_fatal_signal (signum=11) at ./src/global/signal_handler.cc:326 #3 <signal handler called> #4 0x000055ddbd491880 in ?? () #5 0x00007fbd7c040c7f in Messenger::ms_can_fast_dispatch (m=..., this=<optimized out>) at ./src/msg/Messenger.h:650 #6 ProtocolV2::send_message (this=0x55ddbd49a6e0, m=0x7fbd540328a0) at ./src/msg/async/ProtocolV2.cc:414 #7 0x00007fbd7c0115c6 in AsyncConnection::send_message (this=0x55ddbd49a2b0, m=0x7fbd540328a0) at ./src/msg/async/AsyncConnection.cc:566 #8 0x00007fbd7c0c2888 in Connection::send_message2 (m=..., this=0x55ddbd49a2b0) at ./src/msg/Connection.h:126 #9 MonClient::_send_mon_message (this=this@entry=0x55ddbd3ff248, m=...) at ./src/mon/MonClient.cc:678 #10 0x00007fbd7c0c7b36 in MonClient::_renew_subs (this=this@entry=0x55ddbd3ff248) at ./src/mon/MonClient.cc:980 #11 0x00007fbd84e9ea8b in MonClient::renew_subs (this=0x55ddbd3ff248) at ./src/mon/MonClient.h:433 #12 Objecter::_maybe_request_map (this=this@entry=0x55ddbd491880) at ./src/osdc/Objecter.cc:1962 #13 0x00007fbd84e9ed5a in Objecter::maybe_request_map (this=this@entry=0x55ddbd491880) at ./src/osdc/Objecter.cc:1943 #14 0x00007fbd84eba230 in Objecter::ms_handle_reset (this=0x55ddbd491880, con=<optimized out>) at ./src/osdc/Objecter.cc:4401 #15 0x00007fbd7bf7ea19 in Messenger::ms_deliver_handle_reset (con=0x55ddbd4ab990, this=<optimized out>) at ./src/msg/Messenger.h:762 #16 DispatchQueue::entry (this=0x55ddbd498d58) at ./src/msg/DispatchQueue.cc:185 #17 0x00007fbd7c01d7ed in DispatchQueue::DispatchThread::entry (this=<optimized out>) at ./src/msg/DispatchQueue.h:101 #18 0x00007fbd7b8ff6db in start_thread (arg=0x7fbd66ffd700) at pthread_create.c:463 #19 0x00007fbd7a26e88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Core was generated by `rbd rm testimg1'.
Program terminated with signal SIGABRT, Aborted.
#0 raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fc3615a7700 (LWP 10171))]
(gdb) bt
#0 raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x000055f95b728e50 in reraise_fatal (signum=6) at ./src/global/signal_handler.cc:81
#2 handle_fatal_signal (signum=6) at ./src/global/signal_handler.cc:326
#3 <signal handler called>
#4 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#5 0x00007fc36703f801 in __GI_abort () at abort.c:79
#6 0x00007fc367088897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fc3671b5b9a "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#7 0x00007fc36708f90a in malloc_printerr (str=str@entry=0x7fc3671b3cba "corrupted double-linked list") at malloc.c:5350
#8 0x00007fc36709713a in _int_free (have_lock=0, p=<optimized out>, av=0x7fc340000020) at malloc.c:4295
#9 __GI___libc_free (mem=<optimized out>) at malloc.c:3124
#10 0x00007fc371d21009 in librados::CB_AioComplete::operator() (this=<synthetic pointer>) at ./src/librados/AioCompletionImpl.h:151
#11 boost::asio::asio_handler_invoke<librados::CB_AioComplete> (function=<synthetic pointer>...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/handler_invoke_hook.hpp:69
#12 boost_asio_handler_invoke_helpers::invoke<librados::CB_AioComplete, librados::CB_AioComplete> (context=<synthetic pointer>..., function=<synthetic pointer>...)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#13 boost::asio::system_executor::dispatch<librados::CB_AioComplete, std::allocator<void> > (this=<optimized out>, f=<synthetic pointer>) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/impl/system_executor.hpp:39
#14 boost::asio::detail::work_dispatcher<librados::CB_AioComplete>::operator() (this=<synthetic pointer>) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/work_dispatcher.hpp:58
#15 boost::asio::asio_handler_invoke<boost::asio::detail::work_dispatcher<librados::CB_AioComplete> > (function=<synthetic pointer>...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/handler_invoke_hook.hpp:69
#16 boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::work_dispatcher<librados::CB_AioComplete>, boost::asio::detail::work_dispatcher<librados::CB_AioComplete> > (context=<synthetic pointer>...,
function=<synthetic pointer>...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#17 boost::asio::detail::handler_work<boost::asio::detail::work_dispatcher<librados::CB_AioComplete>, boost::asio::system_executor>::complete<boost::asio::detail::work_dispatcher<librados::CB_AioComplete> > (
this=<synthetic pointer>, handler=<synthetic pointer>..., function=<synthetic pointer>...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/handler_work.hpp:82
#18 boost::asio::detail::completion_handler<boost::asio::detail::work_dispatcher<librados::CB_AioComplete> >::do_complete (owner=0x55f95c5413b0, base=0x7fc340003960)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/completion_handler.hpp:70
#19 0x00007fc371d39535 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=0x55f95c5413b0, this=<optimized out>)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/scheduler_operation.hpp:40
#20 boost::asio::detail::strand_service::do_complete (owner=0x55f95c5413b0, base=0x55f95c5ba140, ec=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/impl/strand_service.ipp:168
#21 0x00007fc371d3bc9f in boost::asio::detail::scheduler_operation::complete (bytes_transferred=<optimized out>, ec=..., owner=0x55f95c5413b0, this=<optimized out>)
at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/scheduler_operation.hpp:40
#22 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x55f95c5413b0) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/impl/scheduler.ipp:401
#23 boost::asio::detail::scheduler::run (this=0x55f95c5413b0, ec=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/detail/impl/scheduler.ipp:154
#24 0x00007fc371d3be49 in boost::asio::io_context::run (this=0x55f95c5c01f0) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/impl/io_context.ipp:62
#25 ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const (__closure=<optimized out>) at ./src/common/async/context_pool.h:66
#26 std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#27 std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_result&&, (ceph::async::io_context_pool::start(short)::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#28 std::thread::_Invoker<std::tuple<ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/include/c++/7/thread:234
#29 std::thread::_Invoker<std::tuple<ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() (this=<optimized out>) at /usr/include/c++/7/thread:243
#30 std::thread::_State_impl<std::thread::_Invoker<std::tuple<ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() (this=<optimized out>) at /usr/include/c++/7/thread:186
#31 0x00007fc367a6366f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#32 0x00007fffd28cab90 in ?? ()
#33 0x337cf34aab4f16b8 in ?? ()
#34 0x337ce108cb8b16b8 in ?? ()
#35 0x0000000000000000 in ?? ()
Updated by Jason Dillaman over 4 years ago
- Project changed from rbd to RADOS
Excellent sleuthing -- thanks! I am going to bump this over to the RADOS project since I can't see how this is purely an RBD issue now.
Updated by Jason Dillaman over 4 years ago
@Ilya: does it reproduce when you have injected socket failures disabled? From your initial logs and from the backtrace, it looks like your connections are resetting periodically so I wonder if that's a contributing factor.
Updated by Ilya Dryomov over 4 years ago
Haven't tried without failure injection yet, but it's probably related to ms_inject_socket_failures (and resulting watch errors in some cases), yes.
Updated by Casey Bodley over 4 years ago
- Assignee set to Adam Emerson
one segfault related to watch/notify is fixed in https://github.com/ceph/ceph/pull/31768, but testing in the rgw suite is hitting a lot of other crashes
Updated by Ilya Dryomov over 4 years ago
Ilya Dryomov wrote:
Haven't tried without failure injection yet, but it's probably related to ms_inject_socket_failures (and resulting watch errors in some cases), yes.
With ms_inject_socket_failures = 0, it seems to be holding up.
Updated by Ilya Dryomov over 4 years ago
I managed to grab the stack traces from when it locks up instead of crashing -- also around watch/notify in the face of connection resets.
Actions