Project

General

Profile

Actions
Copy link

Bug #63233

open

mon|client|mds: valgrind reports possible leaks in the MDS

Added by Venky Shankar 6 months ago. Updated 6 months ago.

Status:
New
Priority:
Urgent
Assignee:
Rishabh Dave
Category:
Correctness/Safety
Target version:
Ceph - v19.0.0
% Done:

0%

Source:
Tags:
Backport:
reef,quincy
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/vshankar-2023-10-14_01:51:22-fs-wip-vshankar-testing-20231013.093215-testing-default-smithi/7427332

2023-10-16T04:24:05.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq
2023-10-16T04:24:05.842 DEBUG:teuthology.orchestra.run.smithi062:> sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq
2023-10-16T04:24:05.886 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mds.b.log:  <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.886 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mds.d.log:  <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.887 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mds.f.log:  <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.887 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mon.b.log:  <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.887 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mon.c.log:  <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.978 INFO:teuthology.orchestra.run.smithi031.stdout:/var/log/ceph/valgrind/mds.a.log:  <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.978 INFO:teuthology.orchestra.run.smithi031.stdout:/var/log/ceph/valgrind/mds.c.log:  <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.978 INFO:teuthology.orchestra.run.smithi031.stdout:/var/log/ceph/valgrind/mon.a.log:  <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.979 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.a.log kind   <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.980 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.c.log kind   <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.980 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind   <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.980 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
2023-10-16T04:24:05.980 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.b.log kind   <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.d.log kind   <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.f.log kind   <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind   <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.981 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.b.log
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind   <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.982 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log

Related issues 1 (0 open — 1 closed)

Related to RADOS - Bug #53575: Valgrind reports memory "Leak_PossiblyLost" errors concerning lib64ResolvedNitzan Mordechai

Actions
Actions
Copy link
 #1

Updated by Patrick Donnelly 6 months ago

  • Subject changed from mds: valgrind reports possible leaks in the MDS to mon|client|mds: valgrind reports possible leaks in the MDS

Also the mons/clients.

2023-10-19T14:50:52.435 INFO:tasks.ceph:Checking for errors in any valgrind logs...
2023-10-19T14:50:52.436 DEBUG:teuthology.orchestra.run.smithi105:> sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq                                                       
2023-10-19T14:50:52.438 DEBUG:teuthology.orchestra.run.smithi194:> sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq                                                       
2023-10-19T14:50:52.472 INFO:teuthology.orchestra.run.smithi105.stdout:/var/log/ceph/valgrind/client.0.log:  <kind>Leak_StillReachable</kind>                                                 
2023-10-19T14:50:52.472 INFO:teuthology.orchestra.run.smithi105.stdout:/var/log/ceph/valgrind/mon.a.log:  <kind>Leak_StillReachable</kind>                                                    
2023-10-19T14:50:52.480 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/client.0.log kind   <kind>Leak_StillReachable</kind>                                                                     
2023-10-19T14:50:52.480 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/client.0.log                                                         
2023-10-19T14:50:52.480 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind   <kind>Leak_StillReachable</kind>                                                                        
2023-10-19T14:50:52.481 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log                                                            
2023-10-19T14:50:52.512 INFO:teuthology.orchestra.run.smithi194.stdout:/var/log/ceph/valgrind/mds.f.log:  <kind>Leak_PossiblyLost</kind>                                                      
2023-10-19T14:50:52.512 INFO:teuthology.orchestra.run.smithi194.stdout:/var/log/ceph/valgrind/mon.b.log:  <kind>Leak_StillReachable</kind>                                                    
2023-10-19T14:50:52.512 INFO:teuthology.orchestra.run.smithi194.stdout:/var/log/ceph/valgrind/mon.c.log:  <kind>Leak_StillReachable</kind>                                                    
2023-10-19T14:50:52.513 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.f.log kind   <kind>Leak_PossiblyLost</kind>                                                                          
2023-10-19T14:50:52.513 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind   <kind>Leak_StillReachable</kind>                                                                        
2023-10-19T14:50:52.513 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.b.log                                                            
2023-10-19T14:50:52.513 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind   <kind>Leak_StillReachable</kind>                                                                        
2023-10-19T14:50:52.514 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log

https://pulpito.ceph.com/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432287/

Actions
Copy link
 #2

Updated by Patrick Donnelly 6 months ago

The common frame is:

    <frame>
      <ip>0x4A9B806</ip>
      <obj>/usr/lib/ceph/libceph-common.so.2</obj>
      <fn>_sub_I_65535_0.0</fn>
    </frame

Which indicates some recent change to libceph-common caused this.

Actions
Copy link
 #3

Updated by Milind Changire 6 months ago

  • Assignee set to Rishabh Dave
Actions
Copy link
 #4

Updated by Laura Flores 6 months ago

  • Related to Bug #53575: Valgrind reports memory "Leak_PossiblyLost" errors concerning lib64 added
Actions
Copy link
 #5

Updated by Venky Shankar 6 months ago

  • Priority changed from Normal to Urgent
  • Severity changed from 3 - minor to 2 - major

Bumping prio. Rishabh, PTAL at the earliest.

Actions
Copy link

Also available in: Atom PDF