Bug #23324: delete type mismatch in CephContext teardown - RADOS - Ceph

Actions

Copy link

Bug #23324

closed

delete type mismatch in CephContext teardown

Added by Jeff Layton about 6 years ago. Updated about 6 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Brad Hubbard

Category:

Target version:

% Done:

Source:

Development

Tags:

Backport:

jewel, luminous

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

I've been hunting some memory corruption in ganesha and ran across this. Seems unlikely to be the cause of the crashes I've been seeing, but is something that should be fixed.

12/03/2018 12:47:49 : epoch 5aa6af2c : tleilax.poochiereds.net : ganesha.nfsd-25903[Admin] pseudo_unmount_export :EXPORT :DEBUG :Unmount /cephfs
12/03/2018 12:47:49 : epoch 5aa6af2c : tleilax.poochiereds.net : ganesha.nfsd-25903[Admin] mdcache_exp_release :FSAL :INFO :Releasing Ceph export 100 for /
=================================================================
==25903==ERROR: AddressSanitizer: new-delete-type-mismatch on 0x611000015580 in thread T311:
  object passed to delete has wrong type:
  size of the allocated type:   219 bytes;
  size of the deallocated type: 128 bytes.
    #0 0x7f3a1e15b6d8 in operator delete(void*, unsigned long) (/lib64/libasan.so.4+0xe16d8)
    #1 0x7f3a1222e945 in ceph::logging::Log::~Log() (/usr/lib64/ceph/libceph-common.so.0+0x294945)
    #2 0x7f3a1222ea28 in ceph::logging::Log::~Log() (/usr/lib64/ceph/libceph-common.so.0+0x294a28)
    #3 0x7f3a124027b1 in CephContext::~CephContext() (/usr/lib64/ceph/libceph-common.so.0+0x4687b1)
    #4 0x7f3a12402a30 in CephContext::put() (/usr/lib64/ceph/libceph-common.so.0+0x468a30)
    #5 0x7f3a02fb9c9a in ceph_shutdown (/lib64/libcephfs.so.2+0x14c9a)
    #6 0x7f3a032bdb05 in release /home/jlayton/git/ganesha/src/FSAL/FSAL_CEPH/export.c:75
    #7 0x6b5495 in mdcache_exp_release /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_export.c:187
    #8 0x663d8e in free_export_resources /home/jlayton/git/ganesha/src/support/exports.c:2077
    #9 0x687ac3 in free_export /home/jlayton/git/ganesha/src/support/export_mgr.c:281
    #10 0x68a492 in _put_gsh_export /home/jlayton/git/ganesha/src/support/export_mgr.c:700
    #11 0x68b45d in remove_all_exports /home/jlayton/git/ganesha/src/support/export_mgr.c:833
    #12 0x48b95b in do_shutdown /home/jlayton/git/ganesha/src/MainNFSD/nfs_admin_thread.c:493
    #13 0x48bfa8 in admin_thread /home/jlayton/git/ganesha/src/MainNFSD/nfs_admin_thread.c:533
    #14 0x7f3a1c88b61a in start_thread (/lib64/libpthread.so.0+0x761a)
    #15 0x7f3a1be02c2e in __GI___clone (/lib64/libc.so.6+0x117c2e)

0x611000015580 is located 0 bytes inside of 219-byte region [0x611000015580,0x61100001565b)
allocated by thread T0 here:
    #0 0x7f3a1e15a158 in operator new(unsigned long) (/lib64/libasan.so.4+0xe0158)
    #1 0x7f3a1222f3be in ceph::logging::Log::create_entry(int, int, unsigned long*) (/usr/lib64/ceph/libceph-common.so.0+0x2953be)

Thread T311 created by T0 here:
    #0 0x7f3a1e0b1a2f in pthread_create (/lib64/libasan.so.4+0x37a2f)
    #1 0x47ab89 in nfs_Start_threads /home/jlayton/git/ganesha/src/MainNFSD/nfs_init.c:581
    #2 0x47cee9 in nfs_start /home/jlayton/git/ganesha/src/MainNFSD/nfs_init.c:931
    #3 0x41fd9c in main /home/jlayton/git/ganesha/src/MainNFSD/nfs_main.c:500
    #4 0x7f3a1bd0c009 in __libc_start_main (/lib64/libc.so.6+0x21009)

SUMMARY: AddressSanitizer: new-delete-type-mismatch (/lib64/libasan.so.4+0xe16d8) in operator delete(void*, unsigned long)
==25903==HINT: if you don't care about these errors you may set ASAN_OPTIONS=new_delete_type_mismatch=0
==25903==ABORTING

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by Jeff Layton about 6 years ago

Package in this case is:

librados2-13.0.1-2356.gf2b88f364515.fc27.x86_64

Actions

Copy link

Updated by Greg Farnum about 6 years ago

Project changed from Ceph to CephFS

I don't recognize this from elsewhere and it looks like the kind of issue that could arise from trying to delete something that no longer exists, especially if you already have memory corruption. :)

Actions

Copy link

Updated by Jeff Layton about 6 years ago

It looks more to me like we're allocating an object of one type (Entry) and then casting it to another (Log)? Is there some inheritance there? I don't know the logging code well.

In any case, possibly relevant stackoverflow article:

https://stackoverflow.com/questions/41552966/getting-new-delete-type-mismatch-from-asan

This is getting into deeper C++ voodoo than I know though. If you think it's legit as-is, we can just declare this invalid and move on.

Actions

Copy link

Updated by Brad Hubbard about 6 years ago

Project changed from CephFS to RADOS
Status changed from New to 12
Assignee set to Brad Hubbard
Source set to Development

This has to do with the use of placement new in the overload of Log::create_entry with the expected_size argument. I've resolved this I believe in one part of the code but there are other places where we are getting the new-delete-type-mismatch so more work is needed. Hope to have PR for this tomorrow.

Actions

Copy link