Bug #16685: "saw valgrind issues" in rados run - Ceph - Ceph

Actions

Copy link

Bug #16685

closed

"saw valgrind issues" in rados run

Added by Yuri Weinstein almost 8 years ago. Updated almost 7 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Haomai Wang

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

rados

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Run: http://pulpito.ceph.com/yuriw-2016-07-12_13:07:42-rados-wip-yuri-testing_2016_7_12-distro-basic-smithi/
Jobs: ['310070']['309925']
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2016-07-12_13:07:42-rados-wip-yuri-testing_2016_7_12-distro-basic-smithi/310070/teuthology.log

2016-07-13T02:42:55.964 INFO:teuthology.orchestra.run.smithi024:Running: 'sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/ceph.osdmap /home/ubuntu/cephtest/../*.pid'
2016-07-13T02:42:55.985 INFO:teuthology.orchestra.run.smithi050:Running: 'sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/ceph.osdmap /home/ubuntu/cephtest/../*.pid'
2016-07-13T02:42:56.052 INFO:tasks.ceph:Checking for errors in any valgrind logs...
2016-07-13T02:42:56.052 INFO:teuthology.orchestra.run.smithi024:Running: "sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq" 
2016-07-13T02:42:56.056 INFO:teuthology.orchestra.run.smithi050:Running: "sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq" 
2016-07-13T02:42:56.135 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind   <kind>Leak_StillReachable</kind>
2016-07-13T02:42:56.142 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
2016-07-13T02:42:56.144 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind   <kind>Leak_StillReachable</kind>
2016-07-13T02:42:56.144 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log
2016-07-13T02:42:56.145 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.145 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/mon.c.log
2016-07-13T02:42:56.147 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.0.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.150 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.0.log
2016-07-13T02:42:56.150 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.1.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.151 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.1.log
2016-07-13T02:42:56.153 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.2.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.155 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.2.log
2016-07-13T02:42:56.158 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.158 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/mon.b.log
2016-07-13T02:42:56.159 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.3.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.160 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.3.log
2016-07-13T02:42:56.160 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.4.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.160 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.4.log
2016-07-13T02:42:56.161 DEBUG2016-07-13T02:42:55.964 INFO:teuthology.orchestra.run.smithi024:Running: 'sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/ceph.osdmap /home/ubuntu/cephtest/../*.pid'
2016-07-13T02:42:55.985 INFO:teuthology.orchestra.run.smithi050:Running: 'sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/ceph.osdmap /home/ubuntu/cephtest/../*.pid'
2016-07-13T02:42:56.052 INFO:tasks.ceph:Checking for errors in any valgrind logs...
2016-07-13T02:42:56.052 INFO:teuthology.orchestra.run.smithi024:Running: "sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq" 
2016-07-13T02:42:56.056 INFO:teuthology.orchestra.run.smithi050:Running: "sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq" 
2016-07-13T02:42:56.135 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind   <kind>Leak_StillReachable</kind>
2016-07-13T02:42:56.142 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
2016-07-13T02:42:56.144 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind   <kind>Leak_StillReachable</kind>
2016-07-13T02:42:56.144 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log
2016-07-13T02:42:56.145 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.145 ERROR2016-07-13T02:42:55.964 INFO:teuthology.orchestra.run.smithi024:Running: 'sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/ceph.osdmap /home/ubuntu/cephtest/../*.pid'
2016-07-13T02:42:55.985 INFO:teuthology.orchestra.run.smithi050:Running: 'sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/ceph.osdmap /home/ubuntu/cephtest/../*.pid'
2016-07-13T02:42:56.052 INFO:tasks.ceph:Checking for errors in any valgrind logs...
2016-07-13T02:42:56.052 INFO:teuthology.orchestra.run.smithi024:Running: "sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq" 
2016-07-13T02:42:56.056 INFO:teuthology.orchestra.run.smithi050:Running: "sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq" 
2016-07-13T02:42:56.135 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind   <kind>Leak_StillReachable</kind>
2016-07-13T02:42:56.142 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
2016-07-13T02:42:56.144 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind   <kind>Leak_StillReachable</kind>
2016-07-13T02:42:56.144 ERROR:tasks.ceph:saw valgrind issue   <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log
2016-07-13T02:42:56.145 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.145 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/mon.c.log
2016-07-13T02:42:56.147 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.0.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.150 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.0.log
2016-07-13T02:42:56.150 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.1.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.151 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.1.log
2016-07-13T02:42:56.153 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.2.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.155 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.2.log
2016-07-13T02:42:56.158 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.158 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/mon.b.log
2016-07-13T02:42:56.159 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.3.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.160 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.3.log
2016-07-13T02:42:56.160 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.4.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.160 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.4.log
2016-07-13T02:42:56.161 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.5.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.161 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.5.log:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/mon.c.log
2016-07-13T02:42:56.147 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.0.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.150 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.0.log
2016-07-13T02:42:56.150 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.1.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.151 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.1.log
2016-07-13T02:42:56.153 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.2.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.155 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.2.log
2016-07-13T02:42:56.158 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.158 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/mon.b.log
2016-07-13T02:42:56.159 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.3.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.160 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.3.log
2016-07-13T02:42:56.160 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.4.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.160 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.4.log
2016-07-13T02:42:56.161 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.5.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.161 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.5.log:tasks.ceph:file /var/log/ceph/valgrind/osd.5.log kind   <kind>UninitCondition</kind>
2016-07-13T02:42:56.161 ERROR:tasks.ceph:saw valgrind issue   <kind>UninitCondition</kind> in /var/log/ceph/valgrind/osd.5.log

Actions

Copy link

Updated by Haomai Wang almost 8 years ago

<error>
<unique>0x0</unique>
<tid>1</tid>
<kind>UninitCondition</kind>
<what>Conditional jump or move depends on uninitialised value(s)</what>
<stack>
<frame>
<ip>0x210A465</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>EventCenter::dispatch_event_external(EventCallback*)</fn>
</frame>
<frame>
<ip>0x22BDAAC</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>AsyncConnection::_connect()</fn>
</frame>
<frame>
<ip>0x2100F14</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>AsyncConnection::connect(entity_addr_t const&, int)</fn>
</frame>
<frame>
<ip>0x20FE564</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>AsyncMessenger::create_connect(entity_addr_t const&, int)</fn>
</frame>
<frame>
<ip>0x20FE8A6</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>AsyncMessenger::get_connection(entity_inst_t const&)</fn>
</frame>
<frame>
<ip>0x20575D8</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::_reopen_session(int, std::string)</fn>
</frame>
<frame>
<ip>0x1860A27</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::_reopen_session()</fn>
</frame>
<frame>
<ip>0x2055BE8</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::authenticate(double)</fn>
</frame>
<frame>
<ip>0x17F8D24</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>OSD::init()</fn>
</frame>
<frame>
<ip>0x17D4893</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>main</fn>
</frame>
</stack>
</error>

<error>
<unique>0x1</unique>
<tid>1</tid>
<kind>UninitCondition</kind>
<what>Conditional jump or move depends on uninitialised value(s)</what>
<stack>
<frame>
<ip>0x210A465</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>EventCenter::dispatch_event_external(EventCallback*)</fn>
</frame>
<frame>
<ip>0x22C3E75</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>AsyncConnection::send_keepalive()</fn>
</frame>
<frame>
<ip>0x20579CB</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::_reopen_session(int, std::string)</fn>
</frame>
<frame>
<ip>0x1860A27</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::_reopen_session()</fn>
</frame>
<frame>
<ip>0x2055BE8</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::authenticate(double)</fn>
</frame>
<frame>
<ip>0x17F8D24</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>OSD::init()</fn>
</frame>
<frame>
<ip>0x17D4893</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>main</fn>
</frame>
</stack>
</error>

<error>
<unique>0x2</unique>
<tid>1</tid>
<kind>UninitCondition</kind>
<what>Conditional jump or move depends on uninitialised value(s)</what>
<stack>
<frame>
<ip>0x210A465</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>EventCenter::dispatch_event_external(EventCallback*)</fn>
</frame>
<frame>
<ip>0x22BEB04</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>AsyncConnection::send_message(Message*)</fn>
</frame>
<frame>
<ip>0x20570D9</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::_send_mon_message(Message*, bool)</fn>
</frame>
<frame>
<ip>0x2057AE8</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::_reopen_session(int, std::string)</fn>
</frame>
<frame>
<ip>0x1860A27</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::_reopen_session()</fn>
</frame>
<frame>
<ip>0x2055BE8</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>MonClient::authenticate(double)</fn>
</frame>
<frame>
<ip>0x17F8D24</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>OSD::init()</fn>
</frame>
<frame>
<ip>0x17D4893</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>main</fn>
</frame>
</stack>
</error>

Actions

Copy link

Updated by Kefu Chai almost 8 years ago

Status changed from New to Fix Under Review
Assignee set to Haomai Wang

https://github.com/ceph/ceph/pull/10300

Actions

Copy link

Updated by Yuri Weinstein almost 8 years ago

/a/yuriw-2016-07-25_09:34:28-rados-wip-yuri-testing_2016_7_20-distro-basic-smithi/333016/summary.yaml
/a/yuriw-2016-07-25_09:34:28-rados-wip-yuri-testing_2016_7_20-distro-basic-smithi/333040/summary.yaml

Actions

Copy link

Updated by Yuri Weinstein over 7 years ago

Run: http://pulpito.ceph.com/yuriw-2016-07-29_14:35:52-rados-wip-yuri-testing2_2016_7_30-distro-basic-smithi/
Jobs: 340878, 341165

Actions

Copy link

Updated by Haomai Wang over 7 years ago

Status changed from Fix Under Review to Need More Info

I want to know whether it will reproduce in the current master

Actions

Copy link

Updated by Yuri Weinstein over 7 years ago

/a/yuriw-2016-08-18_19:38:45-rados-wip-yuri-testing_2016_08_18-distro-basic-smithi/372945
/a/yuriw-2016-08-18_19:38:45-rados-wip-yuri-testing_2016_08_18-distro-basic-smithi/373228

Actions

Copy link

Updated by Yuri Weinstein over 7 years ago

/a/yuriw-2016-09-20_14:50:21-rados-wip-yuri-testing2_2016_09_19-distro-basic-smithi/425868/
/a/yuriw-2016-09-20_14:50:21-rados-wip-yuri-testing2_2016_09_19-distro-basic-smithi/425877/

Actions

Copy link

Updated by Greg Farnum almost 7 years ago

Status changed from Need More Info to Closed

We have passed many valgrind runs with the AsyncMessenger in the past 9 months.

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph

Custom queries

Bug #16685

"saw valgrind issues" in rados run

Updated by Haomai Wang almost 8 years ago

Updated by Kefu Chai almost 8 years ago

Updated by Yuri Weinstein almost 8 years ago

Updated by Yuri Weinstein over 7 years ago

Updated by Haomai Wang over 7 years ago

Updated by Yuri Weinstein over 7 years ago

Updated by Yuri Weinstein over 7 years ago

Updated by Greg Farnum almost 7 years ago