Project

General

Profile

Bug #50218

Segmentation fault in radosgw_Main()

Added by Neha Ojha 6 months ago. Updated 4 months ago.

Status:
Closed
Priority:
High
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-04-07T23:10:18.014 INFO:tasks.rgw.c1.client.0.smithi081.stdout:*** Caught signal (Segmentation fault) **
2021-04-07T23:10:18.091 INFO:tasks.rgw.c1.client.0.smithi081.stdout: in thread 7fbf609352c0 thread_name:radosgw
2021-04-07T23:10:18.093 INFO:tasks.rgw.c2.client.0.smithi152.stdout:*** Caught signal (Segmentation fault) **
2021-04-07T23:10:18.093 INFO:tasks.rgw.c2.client.0.smithi152.stdout: in thread 7fe1bf5cb2c0 thread_name:radosgw
2021-04-07T23:10:18.093 INFO:tasks.rgw.c2.client.0.smithi152.stdout: ceph version 16.2.0-63-g0af03c54 (0af03c54623fd04658877fdb3265ff54fb242c3c) pacific (stable)
2021-04-07T23:10:18.093 INFO:tasks.rgw.c2.client.0.smithi152.stdout: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7fe1bdb9d040]
2021-04-07T23:10:18.094 INFO:tasks.rgw.c2.client.0.smithi152.stdout: 2: (std::locale::operator=(std::locale const&)+0x28) [0x56259360ed48]
2021-04-07T23:10:18.094 INFO:tasks.rgw.c2.client.0.smithi152.stdout: 3: (std::ios_base::imbue(std::locale const&)+0x2e) [0x5625936c15de]
2021-04-07T23:10:18.094 INFO:tasks.rgw.c2.client.0.smithi152.stdout: 4: (std::basic_ios<char, std::char_traits<char> >::imbue(std::locale const&)+0x44) [0x562593656b14]
2021-04-07T23:10:18.094 INFO:tasks.rgw.c2.client.0.smithi152.stdout: 5: (std::basic_ostream<char, std::char_traits<char> >& boost::asio::ip::operator<< <char, std::char_traits<char>, boost::asio::ip::tcp>(std::basic_ostream<char, std::char_traits<char> >&, boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> const&)+0x8f) [0x7fe1be67af6f]
2021-04-07T23:10:18.095 INFO:tasks.rgw.c2.client.0.smithi152.stdout: 6: /usr/lib/libradosgw.so.2(+0x49f858) [0x7fe1be65e858]
2021-04-07T23:10:18.095 INFO:tasks.rgw.c2.client.0.smithi152.stdout: 7: (radosgw_Main(int, char const**)+0x362c) [0x7fe1be7e1abc]
2021-04-07T23:10:18.095 INFO:tasks.rgw.c2.client.0.smithi152.stdout: 8: __libc_start_main()
2021-04-07T23:10:18.095 INFO:tasks.rgw.c2.client.0.smithi152.stdout: 9: _start()

rados/upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary}
/a/nojha-2021-04-07_22:43:15-rados-wip-default-mclock-distro-basic-smithi/6027634

Found another report in https://github.com/ceph/ceph/pull/40639#issuecomment-814945648


Related issues

Related to rgw - Bug #50139: RadosGW can't start when upgrade to Pacific (16.2.0) from Octopus (15.2.10) Resolved
Related to rgw - Bug #50269: s3select crash in std::locale Resolved
Duplicated by rgw - Bug #50742: crash: std::locale::operator=(std::locale const&) Duplicate
Duplicated by rgw - Bug #50741: crash: librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*) Won't Fix

History

#1 Updated by Casey Bodley 6 months ago

  • Assignee set to Mark Kogan

#2 Updated by Casey Bodley 6 months ago

  • Related to Bug #50139: RadosGW can't start when upgrade to Pacific (16.2.0) from Octopus (15.2.10) added

#3 Updated by Casey Bodley 6 months ago

  • Priority changed from Normal to High

#4 Updated by Mark Kogan 6 months ago

pasting rgw log from start until the segf

2021-04-07T23:10:17.851+0000 7fbf609352c0  0 deferred set uid:gid to 64045:64045 (ceph:ceph)                                                                                                                                                                                                                                                                                                                                                 
2021-04-07T23:10:17.851+0000 7fbf609352c0  0 ceph version 16.2.0-63-g0af03c54 (0af03c54623fd04658877fdb3265ff54fb242c3c) pacific (stable), process radosgw, pid 18211                                                                                                                                                                                                                                                                        
2021-04-07T23:10:17.851+0000 7fbf609352c0  0 framework: beast                                                                                                                                                                                                                                                                                                                                                                                
2021-04-07T23:10:17.851+0000 7fbf609352c0  0 framework conf key: port, val: 8000                                                                                                                                                                                                                                                                                                                                                             
2021-04-07T23:10:17.851+0000 7fbf609352c0  1 radosgw_Main not setting numa affinity                                                                                                                                                                                                                                                                                                                                                          
2021-04-07T23:10:17.851+0000 7fbf45f76700 20 reqs_thread_entry: start                                                                                                                                                                                                                                                                                                                                                                        
2021-04-07T23:10:17.851+0000 7fbf45775700 10 entry start                                                                                                                                                                                                                                                                                                                                                                                     
2021-04-07T23:10:17.859+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.859+0000 7fbf609352c0 20 rados_obj.operate() r=-2 bl.length=0                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.859+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.859+0000 7fbf609352c0 20 rados_obj.operate() r=-2 bl.length=0                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.859+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.859+0000 7fbf609352c0 20 rados_obj.operate() r=-2 bl.length=0                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.859+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.863+0000 7fbf609352c0 20 rados_obj.operate() r=0 bl.length=46                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.875+0000 7fbf609352c0 20 RGWRados::pool_iterate: got zonegroup_info.bcb436cc-16f5-4046-a493-e59cf3d8458f                                                                                                                                                                                                                                                                                                                 
2021-04-07T23:10:17.875+0000 7fbf609352c0 20 RGWRados::pool_iterate: got zone_info.abb6f73d-5af8-4416-afb8-63fd2d1de295                                                                                                                                                                                                                                                                                                                      
2021-04-07T23:10:17.875+0000 7fbf609352c0 20 RGWRados::pool_iterate: got zone_names.default                                                                                                                                                                                                                                                                                                                                                  
2021-04-07T23:10:17.875+0000 7fbf609352c0 20 RGWRados::pool_iterate: got zonegroups_names.default                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados_obj.operate() r=0 bl.length=46                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados_obj.operate() r=0 bl.length=889                                                                                                                                                                                                                                                                                                                                                           
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados_obj.operate() r=0 bl.length=46                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados_obj.operate() r=0 bl.length=358                                                                                                                                                                                                                                                                                                                                                           
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados_obj.operate() r=-2 bl.length=0                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.879+0000 7fbf609352c0 10  cannot find current period zonegroup using local zonegroup                                                                                                                                                                                                                                                                                                                                     
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados_obj.operate() r=-2 bl.length=0                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados_obj.operate() r=0 bl.length=46                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.879+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados_obj.operate() r=0 bl.length=358                                                                                                                                                                                                                                                                                                                                                           
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 zonegroup default                                                                                                                                                                                                                                                                                                                                                                               
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados_obj.operate() r=-2 bl.length=0                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.883+0000 7fbf609352c0 10 Cannot find current period zone using local zone                                                                                                                                                                                                                                                                                                                                                
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados_obj.operate() r=-2 bl.length=0                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados_obj.operate() r=0 bl.length=46                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados_obj.operate() r=0 bl.length=889                                                                                                                                                                                                                                                                                                                                                           
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 zone default found                                                                                                                                                                                                                                                                                                                                                                              
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados_obj.operate() r=-2 bl.length=0                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados->read ofs=0 len=0                                                                                                                                                                                                                                                                                                                                                                         
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 rados_obj.operate() r=-2 bl.length=0                                                                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 started sync module instance, tier type =                                                                                                                                                                                                                                                                                                                                                       
2021-04-07T23:10:17.883+0000 7fbf609352c0 20 started zone id=abb6f73d-5af8-4416-afb8-63fd2d1de295 (name=default) with tier type =                                                                                                                                                                                                                                                                                                            
2021-04-07T23:10:17.915+0000 7fbf609352c0 20 add_watcher() i=0                                                                                                                                                                                                                                                                                                                                                                               
2021-04-07T23:10:17.915+0000 7fbf609352c0 20 add_watcher() i=1                                                                                                                                                                                                                                                                                                                                                                               
2021-04-07T23:10:17.915+0000 7fbf609352c0 20 add_watcher() i=2                                                                                                                                                                                                                                                                                                                                                                               
2021-04-07T23:10:17.915+0000 7fbf609352c0 20 add_watcher() i=3                                                                                                                                                                                                                                                                                                                                                                               
2021-04-07T23:10:17.915+0000 7fbf609352c0 20 add_watcher() i=4                                                                                                                                                                                                                                                                                                                                                                               
2021-04-07T23:10:17.915+0000 7fbf609352c0 20 add_watcher() i=5                                                                                                                                                                                                                                                                                                                                                                               
2021-04-07T23:10:17.915+0000 7fbf609352c0 20 add_watcher() i=6                                                                                                                                                                                                                                                                                                                                                                               
2021-04-07T23:10:17.919+0000 7fbf609352c0 20 add_watcher() i=7                                                                                                                                                                                                                                                                                                                                                                               
2021-04-07T23:10:17.919+0000 7fbf609352c0  2 all 8 watchers are set, enabling cache                                                                                                                                                                                                                                                                                                                                                          
2021-04-07T23:10:17.923+0000 7fbf3074b700  2 RGWDataChangesLog::ChangesRenewThread: start                                                                                                                                                                                                                                                                                                                                                    
2021-04-07T23:10:17.923+0000 7fbf609352c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1]                                                                                                                                                                                                                                                                                                                        
2021-04-07T23:10:17.923+0000 7fbf609352c0 20 check_secure_mon_conn(): mode 1 is insecure                                                                                                                                                                                                                                                                                                                                                     
2021-04-07T23:10:18.007+0000 7fbf2c743700  2 garbage collection: garbage collection: start                                                                                                                                                                                                                                                                                                                                                   
2021-04-07T23:10:18.007+0000 7fbf2c743700 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=1                                                                                                                                                                                                                                                                                                
2021-04-07T23:10:18.007+0000 7fbf2bf42700  2 object expiration: start                                                                                                                                                                                                                                                                                                                                                                        
2021-04-07T23:10:18.007+0000 7fbf2bf42700 20 processing shard = obj_delete_at_hint.0000000000              
2021-04-07T23:10:18.007+0000 7fbf2af40700 20 reqs_thread_entry: start                                      
2021-04-07T23:10:18.007+0000 7fbf28f3c700 20 reqs_thread_entry: start                                      
2021-04-07T23:10:18.007+0000 7fbf26737700  5 lifecycle: schedule life cycle next start time: Thu Apr  8 00:00:00 2021                                                                                                 
2021-04-07T23:10:18.007+0000 7fbf24733700  5 lifecycle: schedule life cycle next start time: Thu Apr  8 00:00:00 2021                                                                                                 
2021-04-07T23:10:18.007+0000 7fbf2272f700  5 lifecycle: schedule life cycle next start time: Thu Apr  8 00:00:00 2021                                                                                                 
2021-04-07T23:10:18.007+0000 7fbf21f2e700 20 BucketsSyncThread: start                                      
2021-04-07T23:10:18.007+0000 7fbf609352c0 20 init_complete bucket index max shards: 11                     
2021-04-07T23:10:18.007+0000 7fbf2172d700 20 UserSyncThread: start                                         
2021-04-07T23:10:18.007+0000 7fbf609352c0 10 Started notification manager with: 1 workers                  
2021-04-07T23:10:18.007+0000 7fbf1ff2a700 20 INFO: next queues processing will happen at: Wed Apr  7 23:10:48 2021                                                                                                    

2021-04-07T23:10:18.007+0000 7fbf609352c0 20 RGW hostnames:                                                
2021-04-07T23:10:18.007+0000 7fbf609352c0 20 RGW S3website hostnames:                                      
2021-04-07T23:10:18.007+0000 7fbf20f2c700 20 processing logshard = reshard.0000000000                      
2021-04-07T23:10:18.011+0000 7fbf2c743700 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''                                                 
2021-04-07T23:10:18.011+0000 7fbf609352c0  0 framework: beast                                              
2021-04-07T23:10:18.011+0000 7fbf609352c0  0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt                                                                                             
2021-04-07T23:10:18.011+0000 7fbf609352c0  0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key                                                                                             
2021-04-07T23:10:18.011+0000 7fbf609352c0  0 starting handler: beast                                       
2021-04-07T23:10:18.011+0000 7fbf2c743700 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE                                                       
2021-04-07T23:10:18.011+0000 7fbf2c743700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker=''                               
2021-04-07T23:10:18.011+0000 7fbf609352c0 -1 *** Caught signal (Segmentation fault) **                     
 in thread 7fbf609352c0 thread_name:radosgw                                                                

 ceph version 16.2.0-63-g0af03c54 (0af03c54623fd04658877fdb3265ff54fb242c3c) pacific (stable)              
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7fbf5ef07040]                                             
 2: (std::locale::operator=(std::locale const&)+0x28) [0x55eaa400dd48]                                     
 3: (std::ios_base::imbue(std::locale const&)+0x2e) [0x55eaa40c05de]                                       
 4: (std::basic_ios<char, std::char_traits<char> >::imbue(std::locale const&)+0x44) [0x55eaa4055b14]                                                                                                                  
 5: (std::basic_ostream<char, std::char_traits<char> >& boost::asio::ip::operator<< <char, std::char_traits<char>, boost::asio::ip::tcp>(std::basic_ostream<char, std::char_traits<char> >&, boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> const&)+0x8f) [0x7fbf5f9e4f6f]                                                                                                                                                            
 6: /usr/lib/libradosgw.so.2(+0x49f858) [0x7fbf5f9c8858]                                                   
 7: (radosgw_Main(int, char const**)+0x362c) [0x7fbf5fb4babc]                                              
 8: __libc_start_main()                                                                                    
 9: _start()                                                                                               
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.               

#6 Updated by Casey Bodley 6 months ago

crashes seem to be specific to ubuntu? same with https://tracker.ceph.com/issues/49955 and https://tracker.ceph.com/issues/50269

#7 Updated by Neha Ojha 6 months ago

Couple of things I noticed regarding this particular failure:

1. The same tests were passing few days ago (sha1: d2c0f56b9235507e96124d903af6148f1edc34af)

https://pulpito.ceph.com/nojha-2021-04-05_22:22:02-rados-master-distro-basic-smithi/6023041/
https://pulpito.ceph.com/nojha-2021-04-05_22:22:02-rados-master-distro-basic-smithi/6023171/

2. These upgrade tests do not provide any particular distro, so always get pinned to ubuntu 18.04 (I think)

/a/nojha-2021-04-07_22:43:15-rados-wip-default-mclock-distro-basic-smithi/6027634 - failed

2021-04-07T22:53:32.183 INFO:teuthology.task.internal:Checking packages skipped, missing os_type 'None' or ceph hash 'dda3ece3c83703f2c390a51819350b94ddd2814a'
2021-04-07T22:53:32.184 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2021-04-07T22:53:32.193 INFO:teuthology.task.internal:no buildpackages task found
2021-04-07T22:53:32.194 INFO:teuthology.run_tasks:Running task internal.save_config...
2021-04-07T22:53:32.252 INFO:teuthology.task.internal:Saving configuration
2021-04-07T22:53:32.267 INFO:teuthology.run_tasks:Running task internal.check_lock...
2021-04-07T22:53:32.285 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-07T22:53:32.304 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-07 22:45:59.646653', 'description': '/home/teuthworker/archive/nojha-2021-04-07_22:43:15-rados-wip-default-mclock-distro-basic-smithi/6027634', 'locked_by': 'scheduled_nojha@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICsWz3DHHEXISeAW/LbMEFvXnDpCq50DBSyAFs6rR1tP', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi081.front.sepia.ceph.com'}
2021-04-07T22:53:32.325 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-07 22:45:59.646675', 'description': '/home/teuthworker/archive/nojha-2021-04-07_22:43:15-rados-wip-default-mclock-distro-basic-smithi/6027634', 'locked_by': 'scheduled_nojha@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDsHs0fWJHfPfae+WiaSGLtglyU6XkMzS5gXVCJEqJAv', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi152.front.sepia.ceph.com'}

/a/nojha-2021-04-05_22:22:02-rados-master-distro-basic-smithi/6023041/ - passed

2021-04-05T23:04:38.470 INFO:teuthology.task.internal:Checking packages skipped, missing os_type 'None' or ceph hash 'd2c0f56b9235507e96124d903af6148f1edc34af'
2021-04-05T23:04:38.470 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2021-04-05T23:04:38.482 INFO:teuthology.task.internal:no buildpackages task found
2021-04-05T23:04:38.482 INFO:teuthology.run_tasks:Running task internal.save_config...
2021-04-05T23:04:38.519 INFO:teuthology.task.internal:Saving configuration
2021-04-05T23:04:38.536 INFO:teuthology.run_tasks:Running task internal.check_lock...
2021-04-05T23:04:38.601 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-05T23:04:38.625 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-05 22:59:07.866583', 'description': '/home/teuthworker/archive/nojha-2021-04-05_22:22:02-rados-master-distro-basic-smithi/6023041', 'locked_by': 'scheduled_nojha@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGME7diZoaKKtM/QLcnQxs92b5PeiYwpGOqIvLQGkS5M', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi065.front.sepia.ceph.com'}
2021-04-05T23:04:38.649 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-05 22:59:07.866560', 'description': '/home/teuthworker/archive/nojha-2021-04-05_22:22:02-rados-master-distro-basic-smithi/6023041', 'locked_by': 'scheduled_nojha@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILZmz4BFYjLKfFSvggruEGS9rrsSO6ktJ7b1+OA4UwNV', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi081.front.sepia.ceph.com'}

#8 Updated by Mark Kogan 6 months ago

adding addr2line of the crash location

 ceph version 16.2.0-63-g0af03c54 (0af03c54623fd04658877fdb3265ff54fb242c3c) pacific (stable)              
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7fbf5ef07040]                                             
 2: (std::locale::operator=(std::locale const&)+0x28) [0x55eaa400dd48]                                     
 3: (std::ios_base::imbue(std::locale const&)+0x2e) [0x55eaa40c05de]                                       
 4: (std::basic_ios<char, std::char_traits<char> >::imbue(std::locale const&)+0x44) [0x55eaa4055b14]                                                                                                                  
 5: (std::basic_ostream<char, std::char_traits<char> >& boost::asio::ip::operator<< <char, std::char_traits<char>, boost::asio::ip::tcp>(std::basic_ostream<char, std::char_traits<char> >&, boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> const&)+0x8f) [0x7fbf5f9e4f6f]                                                                                                                                                            
 6: /usr/lib/libradosgw.so.2(+0x49f858) [0x7fbf5f9c8858]                                                   
                                ^^^^^^
 7: (radosgw_Main(int, char const**)+0x362c) [0x7fbf5fb4babc]                                              
 8: __libc_start_main()                                                                                    
 9: _start()                                                                                               
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.               
eu-addr2line --debuginfo-path=$(realpath ./usr/lib/debug/) --pretty-print -a -f -i -C -e ./usr/lib/libradosgw.so.2 0x49f858
0x000000000049f858: init at ./src/rgw/rgw_asio_frontend.cc:613:54

vim ./ceph-16.2.0-63-g0af03c54/src/rgw/rgw_asio_frontend.cc +613
...
 507 int AsioFrontend::init()
 508 {
 509   boost::system::error_code ec;
 510   auto& config = conf->get_config_map();
...
 607   ┊ l.acceptor.listen(max_connection_backlog);
 608   ┊ l.acceptor.async_accept(l.socket,
 609   ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ [this, &l] (boost::system::error_code ec) {
 610   ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ accept(l, ec);
 611   ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ ┊ });
 612
-> 613 ┊ ldout(ctx(), 4) << "frontend listening on " << l.endpoint << dendl;
 614   ┊ socket_bound = true;
 615   }
...

#9 Updated by Mark Kogan 6 months ago

analyzing found core at:
http://qa-proxy.ceph.com/teuthology/nojha-2021-04-07_22:43:15-rados-wip-default-mclock-distro-basic-smithi/6027634/remote/smithi081/coredump/1617837018.18211.core.gz

podman run -it --rm --net=host -v ./:/mnt ubuntu:18.04

apt update
apt install gdb

cd /mnt
gdb -ex "set debug-file-directory $(realpath ./deb/usr/lib/debug)" -ex "set solib-search-path $(realpath ./deb/usr/lib)" ./deb/usr/bin/radosgw ./1617837018.18211.core
...
Core was generated by `radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c'.
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fbf609352c0 (LWP 18211))]
Reading symbols from /mnt/deb/usr/lib/libradosgw.so.2.0.0...Reading symbols from /mnt/deb/usr/lib/debug/.build-id/e4/dfdb23462ccef8c2a115d84dc14b9260947a21.debug...done.
done.

(gdb) bt
#0  __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fbf6025d9e3 in reraise_fatal (signum=11) at ./src/global/signal_handler.cc:332
#2  handle_fatal_signal(int) () at ./src/global/signal_handler.cc:332
#3  <signal handler called>
#4  0x000055eaa400dd48 in std::locale::operator=(std::locale const&) ()
#5  0x000055eaa40c05de in std::ios_base::imbue(std::locale const&) ()
#6  0x000055eaa4055b14 in std::basic_ios<char, std::char_traits<char> >::imbue(std::locale const&) ()
#7  0x00007fbf5f9e4f6f in boost::asio::ip::detail::endpoint::to_string[abi:cxx11]() const (this=0x7fff14d07720)
    at ./obj-x86_64-linux-gnu/boost/include/boost/asio/ip/detail/impl/endpoint.ipp:183
#8  boost::asio::ip::operator<< <char, std::char_traits<char>, boost::asio::ip::tcp> (os=..., endpoint=...) at ./obj-x86_64-linux-gnu/boost/include/boost/asio/ip/impl/basic_endpoint.hpp:34
#9  0x00007fbf5f9c8858 in (anonymous namespace)::AsioFrontend::init() () at /usr/include/c++/9/bits/char_traits.h:335
#10 0x00007fbf5fb4babc in radosgw_Main(int, char const**) () at ./src/rgw/rgw_main.cc:608
#11 0x00007fbf5eee9bf7 in __libc_start_main (main=0x55eaa40063b0 <main>, argc=14, argv=0x7fff14d084c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7fff14d084b8) at ../csu/libc-start.c:310
#12 0x000055eaa400672a in _start ()

(gdb) f 9
(gdb) set print pretty on

(gdb) p l.endpoint
$7 = {
  impl_ = {
    data_ = {
      base = {
        sa_family = 2,
        sa_data = "\037@", '\000' <repeats 11 times>
      },
      v4 = {
        sin_family = 2,
        sin_port = 16415,
        sin_addr = {
          s_addr = 0
        },
        sin_zero = "\000\000\000\000\000\000\000" 
      },
      v6 = {
        sin6_family = 2,
        sin6_port = 16415,
        sin6_flowinfo = 0,
        sin6_addr = {
          __in6_u = {
            __u6_addr8 = '\000' <repeats 15 times>,
            __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
            __u6_addr32 = {0, 0, 0, 0}
          }
        },
        sin6_scope_id = 0
      }
    }
  }
}

(gdb) f 7

(gdb) p *this
$10 = {
  data_ = {
    base = {
      sa_family = 2,
      sa_data = "\037@", '\000' <repeats 11 times>
    },
    v4 = {
      sin_family = 2,
      sin_port = 16415,
      sin_addr = {
        s_addr = 0
      },
      sin_zero = "\000\000\000\000\000\000\000" 
    },
    v6 = {
      sin6_family = 2,
      sin6_port = 16415,
      sin6_flowinfo = 0,
      sin6_addr = {
        __in6_u = {
          __u6_addr8 = '\000' <repeats 15 times>,
          __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
          __u6_addr32 = {0, 0, 0, 0}
        }
      },
      sin6_scope_id = 0
    }
  }
}

find . -name endpoint.ipp -exec ls -l {} \;
-rw-r--r-- 1 mkogan mkogan 3499 Apr 22  2020 ./deb/ceph-16.2.0-63-g0af03c54/src/boost/boost/asio/local/detail/impl/endpoint.ipp
-rw-r--r-- 1 mkogan mkogan 2846 Apr 22  2020 ./deb/ceph-16.2.0-63-g0af03c54/src/boost/boost/asio/generic/detail/impl/endpoint.ipp
-rw-r--r-- 1 mkogan mkogan 5850 Apr 22  2020 ./deb/ceph-16.2.0-63-g0af03c54/src/boost/boost/asio/ip/detail/impl/endpoint.ipp

vim ./deb/ceph-16.2.0-63-g0af03c54/src/boost/boost/asio/ip/detail/impl/endpoint.ipp +183
179 #if !defined(BOOST_ASIO_NO_IOSTREAM)                                                                                                                                                      
180 std::string endpoint::to_string() const                                                                                                                                              
181 {                                                                                                                                                                                    
182   std::ostringstream tmp_os;                                                                                                                                                         
->183  tmp_os.imbue(std::locale::classic());                                                                                                                                             
184   if (is_v4())                                                                                                                                                                       
185   ┊ tmp_os << address();                                                                                                                                                             
186   else                                                                                                                                                                               
187   ┊ tmp_os << '[' << address() << ']';                                                                                                                                               
188   tmp_os << ':' << port();                                                                                                                                                           
189                                                                                                                                                                                      
190   return tmp_os.str();                                                                                                                                                               
191 }                                                                                                                                                                                    
192 #endif // !defined(BOOST_ASIO_NO_IOSTREAM)

#10 Updated by Kefu Chai 6 months ago

rados/upgrade/pacific-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary}

/a/kchai-2021-04-12_02:29:32-rados-wip-kefu-testing-2021-04-12-0057-distro-basic-smithi/6037153

crashed on smithi165, which was ubuntu bionic.

16.2.0-90-g50f1821b-1bionic
...
ceph version 16.2.0-90-g50f1821b (50f1821b4caa22e3a2ca880c14f02061498aa47a) pacific (stable)

the pacific build on bionic is built with PPA repo. and we always have "WITH_STATIC_LIBSTDCXX=ON" if the GCC from a PPA repo is used, so the built executable does not depend on the C++ runtime from the updated GCC?

++ use_ppa=true
++ true
++ hookdir=/home/jenkins-build/.pbuilder/hook.d
++ rm -rf /home/jenkins-build/.pbuilder/hook.d
++ mkdir -p /home/jenkins-build/.pbuilder/hook.d
++ echo /home/jenkins-build/.pbuilder/hook.d
+ hookdir=/home/jenkins-build/.pbuilder/hook.d
+ setup_updates_repo /home/jenkins-build/.pbuilder/hook.d
+ local hookdir=/home/jenkins-build/.pbuilder/hook.d
+ [[ ubuntu != ubuntu ]]
+ [[ x86_64 == \x\8\6\_\6\4 ]]
+ cat
+ chmod +x /home/jenkins-build/.pbuilder/hook.d/D04install-updates-repo
+ setup_pbuilder_for_ppa /home/jenkins-build/.pbuilder/hook.d
+ local hookdir=/home/jenkins-build/.pbuilder/hook.d
+ use_ppa
+ case $vers in
+ case $DIST in
+ use_ppa=true
+ true
+ local gcc_ver=7
+ '[' bionic = bionic ']'
+ gcc_ver=9
+ setup_pbuilder_for_new_gcc /home/jenkins-build/.pbuilder/hook.d 9
...
-- The CXX compiler identification is GNU 9.3.0
-- The C compiler identification is GNU 9.3.0

probably we could try to test on focal so no 3rd party GCC is involved?

https://github.com/ceph/ceph/pull/40803 is created in hope to address (work around) this issue.

#11 Updated by Casey Bodley 5 months ago

  • Related to Bug #50269: s3select crash in std::locale added

#12 Updated by Casey Bodley 5 months ago

  • Status changed from New to Resolved

#13 Updated by Casey Bodley 5 months ago

  • Duplicated by Bug #50742: crash: std::locale::operator=(std::locale const&) added

#14 Updated by Casey Bodley 4 months ago

  • Duplicated by Bug #50741: crash: librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::ObjectWriteOperation*) added

#15 Updated by Yaarit Hatuka 4 months ago

  • Status changed from Resolved to Closed

Not a Ceph bug, changing to Closed.

Also available in: Atom PDF