Project

General

Profile

Bug #23324

delete type mismatch in CephContext teardown

Added by Jeff Layton over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
03/12/2018
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
jewel, luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:

Description

I've been hunting some memory corruption in ganesha and ran across this. Seems unlikely to be the cause of the crashes I've been seeing, but is something that should be fixed.

12/03/2018 12:47:49 : epoch 5aa6af2c : tleilax.poochiereds.net : ganesha.nfsd-25903[Admin] pseudo_unmount_export :EXPORT :DEBUG :Unmount /cephfs
12/03/2018 12:47:49 : epoch 5aa6af2c : tleilax.poochiereds.net : ganesha.nfsd-25903[Admin] mdcache_exp_release :FSAL :INFO :Releasing Ceph export 100 for /
=================================================================
==25903==ERROR: AddressSanitizer: new-delete-type-mismatch on 0x611000015580 in thread T311:
  object passed to delete has wrong type:
  size of the allocated type:   219 bytes;
  size of the deallocated type: 128 bytes.
    #0 0x7f3a1e15b6d8 in operator delete(void*, unsigned long) (/lib64/libasan.so.4+0xe16d8)
    #1 0x7f3a1222e945 in ceph::logging::Log::~Log() (/usr/lib64/ceph/libceph-common.so.0+0x294945)
    #2 0x7f3a1222ea28 in ceph::logging::Log::~Log() (/usr/lib64/ceph/libceph-common.so.0+0x294a28)
    #3 0x7f3a124027b1 in CephContext::~CephContext() (/usr/lib64/ceph/libceph-common.so.0+0x4687b1)
    #4 0x7f3a12402a30 in CephContext::put() (/usr/lib64/ceph/libceph-common.so.0+0x468a30)
    #5 0x7f3a02fb9c9a in ceph_shutdown (/lib64/libcephfs.so.2+0x14c9a)
    #6 0x7f3a032bdb05 in release /home/jlayton/git/ganesha/src/FSAL/FSAL_CEPH/export.c:75
    #7 0x6b5495 in mdcache_exp_release /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_export.c:187
    #8 0x663d8e in free_export_resources /home/jlayton/git/ganesha/src/support/exports.c:2077
    #9 0x687ac3 in free_export /home/jlayton/git/ganesha/src/support/export_mgr.c:281
    #10 0x68a492 in _put_gsh_export /home/jlayton/git/ganesha/src/support/export_mgr.c:700
    #11 0x68b45d in remove_all_exports /home/jlayton/git/ganesha/src/support/export_mgr.c:833
    #12 0x48b95b in do_shutdown /home/jlayton/git/ganesha/src/MainNFSD/nfs_admin_thread.c:493
    #13 0x48bfa8 in admin_thread /home/jlayton/git/ganesha/src/MainNFSD/nfs_admin_thread.c:533
    #14 0x7f3a1c88b61a in start_thread (/lib64/libpthread.so.0+0x761a)
    #15 0x7f3a1be02c2e in __GI___clone (/lib64/libc.so.6+0x117c2e)

0x611000015580 is located 0 bytes inside of 219-byte region [0x611000015580,0x61100001565b)
allocated by thread T0 here:
    #0 0x7f3a1e15a158 in operator new(unsigned long) (/lib64/libasan.so.4+0xe0158)
    #1 0x7f3a1222f3be in ceph::logging::Log::create_entry(int, int, unsigned long*) (/usr/lib64/ceph/libceph-common.so.0+0x2953be)

Thread T311 created by T0 here:
    #0 0x7f3a1e0b1a2f in pthread_create (/lib64/libasan.so.4+0x37a2f)
    #1 0x47ab89 in nfs_Start_threads /home/jlayton/git/ganesha/src/MainNFSD/nfs_init.c:581
    #2 0x47cee9 in nfs_start /home/jlayton/git/ganesha/src/MainNFSD/nfs_init.c:931
    #3 0x41fd9c in main /home/jlayton/git/ganesha/src/MainNFSD/nfs_main.c:500
    #4 0x7f3a1bd0c009 in __libc_start_main (/lib64/libc.so.6+0x21009)

SUMMARY: AddressSanitizer: new-delete-type-mismatch (/lib64/libasan.so.4+0xe16d8) in operator delete(void*, unsigned long)
==25903==HINT: if you don't care about these errors you may set ASAN_OPTIONS=new_delete_type_mismatch=0
==25903==ABORTING


Related issues

Copied to RADOS - Backport #23412: luminous: delete type mismatch in CephContext teardown Resolved
Copied to RADOS - Backport #23413: jewel: delete type mismatch in CephContext teardown Resolved

History

#1 Updated by Jeff Layton over 1 year ago

Package in this case is:

librados2-13.0.1-2356.gf2b88f364515.fc27.x86_64

#2 Updated by Greg Farnum over 1 year ago

  • Project changed from Ceph to fs

I don't recognize this from elsewhere and it looks like the kind of issue that could arise from trying to delete something that no longer exists, especially if you already have memory corruption. :)

#3 Updated by Jeff Layton over 1 year ago

It looks more to me like we're allocating an object of one type (Entry) and then casting it to another (Log)? Is there some inheritance there? I don't know the logging code well.

In any case, possibly relevant stackoverflow article:

https://stackoverflow.com/questions/41552966/getting-new-delete-type-mismatch-from-asan

This is getting into deeper C++ voodoo than I know though. If you think it's legit as-is, we can just declare this invalid and move on.

#4 Updated by Brad Hubbard over 1 year ago

  • Project changed from fs to RADOS
  • Status changed from New to Verified
  • Assignee set to Brad Hubbard
  • Source set to Development

This has to do with the use of placement new in the overload of Log::create_entry with the expected_size argument. I've resolved this I believe in one part of the code but there are other places where we are getting the new-delete-type-mismatch so more work is needed. Hope to have PR for this tomorrow.

#5 Updated by Kefu Chai over 1 year ago

  • Backport set to jewel, luminous

#6 Updated by Brad Hubbard over 1 year ago

  • Status changed from Verified to In Progress

#7 Updated by Kefu Chai over 1 year ago

  • Status changed from In Progress to Pending Backport

#8 Updated by Nathan Cutler over 1 year ago

  • Copied to Backport #23412: luminous: delete type mismatch in CephContext teardown added

#9 Updated by Nathan Cutler over 1 year ago

  • Copied to Backport #23413: jewel: delete type mismatch in CephContext teardown added

#10 Updated by Nathan Cutler about 1 year ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF