https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2011-06-03T10:23:39ZCeph Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=42162011-06-03T10:23:39ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> set to <i>v0.30</i></li></ul><p>Wanted to check in on this one. Are you still seeing this problem? When the processes are zombies, are there any btrfs messages in dmesg? (Usually it's caused by a syscall that's blocked in the kernel and the process can't be cleaned up until it returns...)</p> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=42332011-06-06T04:35:55ZWido den Hollanderwido@42on.com
<ul></ul><p>I've been away for some time, so haven't got a chance to hunt this one further down. I'm back now, so I'll see if I can reproduce it.</p>
<p>As far as I remember, there were some btrfs messages in the dmesg, not sure which one though.</p> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=42452011-06-08T07:25:27ZWido den Hollanderwido@42on.com
<ul></ul><p>I'm just upgrading my cluster, I see a Zombie from May 26th, running v0.28.1 and 2.6.38.2.</p>
<p>I'll see if I can reproduce it with the latest master.</p> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=44002011-06-20T12:06:43ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.30</i> to <i>v0.31</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=44602011-06-21T13:01:23ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li></ul><p>i'm going to close this one. if you see a zombie with a btrfs error in dmesg, it's a btrfs problem. if you don't, let's reopen!</p> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=45412011-06-21T20:40:30ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.31</i> to <i>v0.32</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=46412011-06-29T04:05:37ZWido den Hollanderwido@42on.com
<ul></ul><p>I'm seeing it again. Trying to upgrade from v0.29.1 to v0.30.</p>
<p>My cluster was bouncing all over again, "wrongly marked me down" was everywhere, so I thought I'd give v0.30 a try.</p>
<p>The upgrade is pretty straight forward:</p>
<pre>dsh -g osd-mdb "service ceph stop osd; sleep 10; dpkg -i /tmp/*.deb; service ceph start osd"</pre>
<p>On for example atom0 this caused:</p>
<pre>root@atom0:~# ps aux|grep cosd
root 13346 13.6 0.0 0 0 ? Zsl Jun23 1185:08 [cosd] <defunct>
root 15570 11.5 0.0 0 0 ? Zsl Jun23 1007:46 [cosd] <defunct>
root 26095 0.0 0.0 7676 824 pts/0 S+ 13:02 0:00 grep --color=auto cosd
root@atom0:~#</pre>
<p>Top shows them eating about a 100% CPU.</p>
<p>No btrfs messages, but what I do see is that two "btrfs-cleaner" cleaner processes are pretty busy, could it be that the OSD is still waiting for some stuff from btrfs? I guess so?</p> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=46442011-06-29T09:09:52ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>4</i></li><li><strong>Target version</strong> changed from <i>v0.32</i> to <i>v0.31</i></li></ul><p>If you can reproduce this, the next step is to figure out what the kernel threads are doing. Do several rounds of</p>
<pre><code>echo t > /proc/sysrq-trigger</code></pre>
<p>every few seconds and then capture the kernel log output and attach here.</p>
<p>'w' would dump only threads in uninterruptible state (which I think should capture the cosd threads), but 't' will dump everything and tell us what btrfs-cleaner is doing that is blocking the cosd threads.</p>
<p>Thanks!</p> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=47012011-07-05T09:36:50ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.31</i> to <i>v0.32</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=49932011-07-23T20:36:19ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.32</i> to <i>v0.33</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=52062011-08-08T09:18:44ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.33</i> to <i>v0.34</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=53302011-08-08T11:45:28ZSage Weilsage@newdream.net
<ul><li><strong>translation missing: en.field_position</strong> set to <i>1</i></li><li><strong>translation missing: en.field_position</strong> changed from <i>1</i> to <i>808</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=55252011-08-20T11:08:38ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.34</i> to <i>v0.35</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=60992011-09-06T21:49:15ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.35</i> to <i>v0.36</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=62702011-09-07T12:07:22ZSage Weilsage@newdream.net
<ul><li><strong>translation missing: en.field_position</strong> deleted (<del><i>860</i></del>)</li><li><strong>translation missing: en.field_position</strong> set to <i>905</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=64152011-09-25T14:14:57ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.36</i> to <i>v0.37</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=65792011-10-09T20:39:53ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.37</i> to <i>v0.38</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=66362011-10-09T20:42:34ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>4</i> to <i>Can't reproduce</i></li></ul> Ceph - Bug #1032: osd: Marked down and become zombies after killinghttps://tracker.ceph.com/issues/1032?journal_id=66372011-10-09T20:42:51ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.38</i> to <i>v0.37</i></li><li><strong>translation missing: en.field_position</strong> deleted (<del><i>938</i></del>)</li><li><strong>translation missing: en.field_position</strong> set to <i>1</i></li><li><strong>translation missing: en.field_position</strong> changed from <i>1</i> to <i>943</i></li></ul>