Project

General

Profile

Bug #59269

test_librgw_file.sh crashes: src/tcmalloc.cc:332] Attempt to free invalid pointer 0x55e8173eebd0

Added by Casey Bodley 8 months ago. Updated 8 months ago.

Status:
Pending Backport
Priority:
Urgent
Assignee:
Casey Bodley
Target version:
-
% Done:

0%

Source:
Tags:
tcmalloc focal backport_processed
Backport:
pacific quincy reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
51068
Crash signature (v1):
Crash signature (v2):

Description

happens consistently against ubuntu 20.04: https://pulpito.ceph.com/cbodley-2023-03-30_18:07:30-rgw:verify-main-distro-default-smithi/

2023-03-30T18:51:18.061 INFO:tasks.workunit.client.0.smithi157.stdout:phase 1.1
2023-03-30T18:51:18.084 INFO:tasks.workunit.client.0.smithi157.stderr:src/tcmalloc.cc:332] Attempt to free invalid pointer 0x55e8173eebd0
2023-03-30T18:51:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 134
2023-03-30T18:51:18.133 INFO:tasks.workunit.client.0.smithi157.stderr:Aborted (core dumped)

i was able to reproduce lots of crashes like this under vstart.sh in a focal vm - not just with ceph_test_librgw_file_nfsns, but also ceph-mon, ceph-osd, and radosgw. they all crash immediately on startup with this stack trace:

(gdb) bt                                                                                                                                    
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50                                                                   
#1  0x00007fffd0e1a859 in __GI_abort () at abort.c:79                                                                                       
#2  0x00007fffd21391d2 in ?? () from /lib/x86_64-linux-gnu/libtcmalloc.so.4                                                                 
#3  0x00007fffd213aca9 in ?? () from /lib/x86_64-linux-gnu/libtcmalloc.so.4                                                                 
#4  0x00007fffd214fe1d in MallocExtension::Initialize() () from /lib/x86_64-linux-gnu/libtcmalloc.so.4                                      
#5  0x00007fffd2139b1e in ?? () from /lib/x86_64-linux-gnu/libtcmalloc.so.4                                                                 
#6  0x00007ffff7fe0b9a in call_init (l=<optimized out>, argc=argc@entry=8, argv=argv@entry=0x7fffffffdff8, env=env@entry=0x7fffffffe040)    
    at dl-init.c:72                                                                                                                         
#7  0x00007ffff7fe0ca1 in call_init (env=0x7fffffffe040, argv=0x7fffffffdff8, argc=8, l=<optimized out>) at dl-init.c:30                    
#8  _dl_init (main_map=0x7ffff7ffe190, argc=8, argv=0x7fffffffdff8, env=0x7fffffffe040) at dl-init.c:119                                    
#9  0x00007ffff7fd013a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2                                                                
#10 0x0000000000000008 in ?? ()                                                                                                             
#11 0x00007fffffffe361 in ?? ()                                                                                                             
#12 0x00007fffffffe387 in ?? ()                                                                                                             
#13 0x00007fffffffe38e in ?? ()                                                                                                             
#14 0x00007fffffffe391 in ?? ()                                                                                                             
#15 0x00007fffffffe3b4 in ?? ()                                                                                                             
#16 0x00007fffffffe3b7 in ?? ()                                                                                                             
#17 0x00007fffffffe3b9 in ?? ()                                                                                                             
#18 0x00007fffffffe3d8 in ?? ()                                                                                                             
#19 0x0000000000000000 in ?? ()

Related issues

Related to CephFS - Bug #58219: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) [Command crashed: 'ceph-dencoder type JournalPointer import - decode dump_json'] Resolved
Copied to rgw - Backport #59492: pacific: test_librgw_file.sh crashes: src/tcmalloc.cc:332] Attempt to free invalid pointer 0x55e8173eebd0 New
Copied to rgw - Backport #59493: reef: test_librgw_file.sh crashes: src/tcmalloc.cc:332] Attempt to free invalid pointer 0x55e8173eebd0 Resolved
Copied to rgw - Backport #59494: quincy: test_librgw_file.sh crashes: src/tcmalloc.cc:332] Attempt to free invalid pointer 0x55e8173eebd0 New

History

#1 Updated by Casey Bodley 8 months ago

  • Related to Bug #58219: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) [Command crashed: 'ceph-dencoder type JournalPointer import - decode dump_json'] added

#2 Updated by Casey Bodley 8 months ago

Casey Bodley wrote:

i was able to reproduce lots of crashes like this under vstart.sh in a focal vm - not just with ceph_test_librgw_file_nfsns, but also ceph-mon, ceph-osd, and radosgw. they all crash immediately on startup with this stack trace:

after rebuilding without the cmake options WITH_ASAN and WITH_UBSAN, the other crashes went away but ceph_test_librgw_file_nfsns still crashes on startup

#3 Updated by Casey Bodley 8 months ago

  • Assignee set to Casey Bodley

#4 Updated by Casey Bodley 8 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 51068

#5 Updated by Casey Bodley 8 months ago

  • Status changed from Fix Under Review to Pending Backport

#6 Updated by Backport Bot 8 months ago

  • Copied to Backport #59492: pacific: test_librgw_file.sh crashes: src/tcmalloc.cc:332] Attempt to free invalid pointer 0x55e8173eebd0 added

#7 Updated by Backport Bot 8 months ago

  • Copied to Backport #59493: reef: test_librgw_file.sh crashes: src/tcmalloc.cc:332] Attempt to free invalid pointer 0x55e8173eebd0 added

#8 Updated by Backport Bot 8 months ago

  • Copied to Backport #59494: quincy: test_librgw_file.sh crashes: src/tcmalloc.cc:332] Attempt to free invalid pointer 0x55e8173eebd0 added

#9 Updated by Backport Bot 8 months ago

  • Tags changed from tcmalloc focal to tcmalloc focal backport_processed

Also available in: Atom PDF