https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2016-03-17T18:11:57ZCeph Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=677392016-03-17T18:11:57ZDan van der Ster
<ul></ul><p>Btw, no additional errors logged on the system.</p> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=677892016-03-18T08:57:27ZDan van der Ster
<ul></ul><p>I found rhbz#1245144, also rhbz#1283112, which are related. We are already running parted-3.1-23.el7.x86_64, which has the supposed fix, but clearly it's still racey, or we're not using partprobe correctly. There is a comment in the bz:</p>
<p>"Note that if you are calling parted multiple times from a script and not checking for device nodes to appear/disappear/whatever you will end up in the same situation. It is best to combine all the commands into a single parted call, or check for the expected changes between the calls."</p>
<p>Maybe we should check for the partition's existence before (unnecessarily) calling partprobe?</p> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=677902016-03-18T09:14:31ZDan van der Ster
<ul></ul><p>I installed parted-3.2-16.fc22.x86_64 on this machine and found it is now 100% reliable:</p>
<ol>
<li>rpm -q parted<br />parted-3.2-16.fc22.x86_64</li>
</ol>
<ol>
<li>/usr/sbin/sgdisk --new=2:0:20480M --change-name=2:'ceph journal' --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sde<br />The operation has completed successfully.</li>
<li>partprobe /dev/sde</li>
<li>partprobe /dev/sde</li>
<li>partprobe /dev/sde</li>
<li>partprobe /dev/sde</li>
<li>for i in `seq 100`; do partprobe /dev/sde; done
#</li>
</ol> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=687632016-04-06T14:43:28ZBen Meekhofbmeekhof@umich.edu
<ul></ul><p>I had the same issue with parted-3.1-23.el7.x86_64 (Scientific Linux 7.2). Runs of ceph-disk prepare would fail randomly with the same error from partprobe 'Device or resource busy', and the same error is repeated just by running 'partprobe' with any ceph OSD or Journal partitions in place. There was no consistency to the error. In repeated runs sometimes partprobe would return fine, sometimes not. Presumably the issue is triggered by the udev rules that Ceph (Infernalis) installs in /usr/lib/udev/rules.d/60-ceph-partuuid-workaround.rules</p>
<p>The issue was also resolved for me by installing parted-3.2-16.fc22.x86_64. I've seen no further indication of a problem in running through a batch of 'ceph-disk prepare'.</p> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=712592016-05-23T07:19:12ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Duplicated by</strong> <i><a class="issue tracker-1 status-10 priority-6 priority-high2 closed" href="/issues/15918">Bug #15918</a>: ceph-disk prepare: occasional partprobe failed on CentOS 7/RHEL 7 with parted < 3.2.16</i> added</li></ul> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=712602016-05-23T07:19:44ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li><li><strong>Backport</strong> set to <i>jewel</i></li></ul> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=712612016-05-23T07:30:16ZLoïc Dacharyloic@dachary.org
<ul></ul><ul>
<li><a class="external" href="http://ftp.scientificlinux.org/linux/scientific/7.2/SRPMS/vendor/parted-3.1-23.el7.src.rpm">http://ftp.scientificlinux.org/linux/scientific/7.2/SRPMS/vendor/parted-3.1-23.el7.src.rpm</a></li>
<li><a class="external" href="https://dl.fedoraproject.org/pub/fedora/linux/updates/22/SRPMS/p/parted-3.2-10.fc22.src.rpm">https://dl.fedoraproject.org/pub/fedora/linux/updates/22/SRPMS/p/parted-3.2-10.fc22.src.rpm</a></li>
<li><a class="external" href="https://dl.fedoraproject.org/pub/fedora/linux/updates/23/SRPMS/p/parted-3.2-18.fc23.src.rpm">https://dl.fedoraproject.org/pub/fedora/linux/updates/23/SRPMS/p/parted-3.2-18.fc23.src.rpm</a></li>
</ul>
<ul>
<li>Reviewed source packages patch differences between 3.1-23 and 3.2-18 shows <br /> they both have
<ul>
<li>0033-libparted-Use-read-only-when-probing-devices-on-linu.patch</li>
<li>0020-tests-Use-wait_for_dev_to_-functions.patch<br /> but only 3.2-18 has</li>
<li>0026-tests-Add-udevadm-settle-to-wait_for_-loop-1260664.patch</li>
</ul></li>
</ul>
<ul>
<li>The biggest related change is the <a href="http://git.savannah.gnu.org/cgit/parted.git/commit/?id=5910f1bc983fbab31f9ec86a7166feee4869c21a" class="external">algorithm used to modify, add, and remove partitions</a> with a number of followups (as noted by Brian C. Lane).</li>
</ul>
<ul>
<li>Mailing list thread "parted behavior change between 3.1 and 3.2"</li>
</ul>
<pre>
[ubuntu@mira061 ~]$ sudo yum list installed | grep parted
parted.x86_64 3.1-23.el7 @anaconda
[ubuntu@mira061 ~]$ /usr/sbin/sgdisk --new=2:0:100M --change-name=2:'ceph journal' --mbrtogpt -- /dev/sdb
***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format.
***************************************************************
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
[ubuntu@mira061 ~]$ sudo partprobe /dev/sdb
[ubuntu@mira061 ~]$ ls -l /dev/sdb*
brw-rw----. 1 root disk 8, 16 May 23 08:21 /dev/sdb
brw-rw----. 1 root disk 8, 18 May 23 08:21 /dev/sdb2
[ubuntu@mira061 ~]$ /usr/sbin/sgdisk --new=3:101M:200M --change-name=3:'ceph journal' --mbrtogpt -- /dev/sdb
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
[ubuntu@mira061 ~]$ ls -l /dev/sdb*
brw-rw----. 1 root disk 8, 16 May 23 08:22 /dev/sdb
brw-rw----. 1 root disk 8, 18 May 23 08:22 /dev/sdb2
brw-rw----. 1 root disk 8, 19 May 23 08:22 /dev/sdb3
[ubuntu@mira061 ~]$ sudo mkfs /dev/sdb2
[ubuntu@mira061 ~]$ sudo mount /dev/sdb2 /mnt
[ubuntu@mira061 ~]$ /usr/sbin/sgdisk --new=4:201M:300M --change-name=4:'ceph journal' --mbrtogpt -- /dev/sdb
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
[ubuntu@mira061 ~]$ ls -l /dev/sdb*
brw-rw----. 1 root disk 8, 16 May 23 08:24 /dev/sdb
brw-rw----. 1 root disk 8, 18 May 23 08:24 /dev/sdb2
brw-rw----. 1 root disk 8, 19 May 23 08:24 /dev/sdb3
[ubuntu@mira061 ~]$ sudo partprobe /dev/sdb
[ubuntu@mira061 ~]$ ls -l /dev/sdb*
brw-rw----. 1 root disk 8, 16 May 23 08:24 /dev/sdb
brw-rw----. 1 root disk 8, 18 May 23 08:24 /dev/sdb2
brw-rw----. 1 root disk 8, 19 May 23 08:24 /dev/sdb3
brw-rw----. 1 root disk 8, 20 May 23 08:24 /dev/sdb4
[ubuntu@mira061 ~]$ uname -a
Linux mira061 4.6.0-rc3-ceph-15705-gac8ec84 #1 SMP Fri May 20 04:17:10 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux
</pre>
<p>Same as above with <strong>Linux mira061 3.10.0-327.el7.x86_64 <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: gpf in tcp_sendpage (Closed)" href="https://tracker.ceph.com/issues/1">#1</a> SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux</strong></p> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=713822016-05-25T10:07:02ZLoïc Dacharyloic@dachary.org
<ul></ul><p>After running (script provided by Daniel H. )</p>
<pre>
journal_dev="/dev/vdb"
#osd_devs="/dev/vdc /dev/vdd /dev/vde"
osd_devs="/dev/vdc"
date '+== [%H:%M:%S] ======================================================='
for dev in ${osd_devs} ${journal_dev}; do
echo "RUN: sgdisk --zap-all --clear --mbrtogpt -g -- ${dev}"
sgdisk --zap-all --clear --mbrtogpt -g -- ${dev} 2>&1
echo "rcode=$?"
echo
done
udevadm settle
echo
sleep 2
echo "RUN: partprobe"
partprobe 2>&1
echo "rcode=$?"
udevadm settle
for osd_dev in ${osd_devs}; do
echo "RUN: ceph-disk prepare --cluster ceph ${osd_dev} ${journal_dev}"
ceph-disk --verbose prepare --cluster ceph ${osd_dev} ${journal_dev} 2>&1 || exit 1
echo "rcode=$?"
echo
done
echo
</pre>
<p>On a <br /><pre>
# lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.2 (Maipo)
Release: 7.2
Codename: Maipo
</pre></p>
<pre><code>/usr/sbin/partprobe /dev/vdc</code></pre>
<p>always fails with</p>
<pre>
Error: Error informing the kernel about modifications to partition /dev/vdc1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/vdc1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.
Error: Failed to add partition 1 (Device or resource busy)
</pre>
<ul>
<li>grep vdc /proc/mounts has nothing</li>
<li>fuser /dev/vdc /dev/vdc1 has nothing</li>
<li>lsof -n | grep vdc has nothing</li>
<li>ls -l /dev/mapper is empty</li>
</ul> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=714212016-05-25T13:46:31ZIlya Dryomov
<ul></ul><p>It doesn't stay busy, does it? It's clearly a race between partprobe<br />and udev. I'm guessing something like this:</p>
<p>- partprobe triggers a rescan of the partition table in the kernel<br />- a change uevent is generated by the kernel<br />- if partprobe gets to the device first (for whatever reason it seems<br /> to open/close it multiple times during "partprobe <dev>"), udev backs<br /> off and partprobe's BLKPG_ADD_PARTITION succeeds<br />- if udev gets to the device first, we've got a problem...</p>
<p>I haven't worked through the particulars, but I think the above<br />captures it. It could be that even if udev manages to grab the device<br />and issue BLKRRPART, but partprobe for whatever reason gets delayed<br />long enough for udev to finish, things work out in the end too.</p>
<p>Do you want to get the bottom of it? If it's fixed in partprobe 3.2,<br />what exactly are you after here - a workaround for 3.1?</p> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=714282016-05-25T14:43:57ZLoïc Dacharyloic@dachary.org
<ul></ul><blockquote>
<p>Do you want to get the bottom of it? If it's fixed in partprobe 3.2,<br />what exactly are you after here - a workaround for 3.1?</p>
</blockquote>
<p>Yes, a workaround is required because upgrading to 3.2 may not be possible on RHEL 7.2 (and other platforms maybe).</p> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=714292016-05-25T14:44:53ZLoïc Dacharyloic@dachary.org
<ul></ul><blockquote>
<p>It doesn't stay busy, does it?</p>
</blockquote>
<p>It stays busy forever.</p> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=714332016-05-25T15:12:15ZIlya Dryomov
<ul></ul><p>Are you sure? If you repeat the partprobe again after a couple of seconds or try it a few times at random, at least one of the invocations has got to succeed.<br />Maybe try to invoke partprobe under strace to mess up with timing? It's got to be a race and a 100% hit rate is <strong>very</strong> unlikely...</p> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=714442016-05-25T17:12:47ZLoïc Dacharyloic@dachary.org
<ul></ul><blockquote>
<p>Are you sure? If you repeat the partprobe again after a couple of seconds or try it a few times at random, at least one of the invocations has got to succeed.</p>
</blockquote>
<p>You are correct, it does not always fail. I was (un)lucky enough in the past hours to always hit the case when it fails. But running a loop clearly shows that it fails intermittently.</p>
<pre>
# for i in $(seq 1 9) ; do echo partprobe ; /usr/sbin/partprobe /dev/vdc ; sleep 2 ; done
partprobe
Error: Error informing the kernel about modifications to partition /dev/vdc1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/vdc1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.
Error: Failed to add partition 1 (Device or resource busy)
partprobe
partprobe
partprobe
Error: Error informing the kernel about modifications to partition /dev/vdc1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/vdc1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.
Error: Failed to add partition 1 (Device or resource busy)
partprobe
partprobe
partprobe
partprobe
partprobe
</pre> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=714462016-05-25T17:23:58ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Fix Under Review</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/9330">https://github.com/ceph/ceph/pull/9330</a></p> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=738562016-07-04T13:56:10ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li></ul> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=738682016-07-04T17:23:10ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/16586">Backport #16586</a>: jewel: partprobe intermittent issues during ceph-disk prepare</i> added</li></ul> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=742632016-07-12T12:43:40ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Duplicated by</strong> <i><a class="issue tracker-1 status-10 priority-5 priority-high3 closed" href="/issues/13984">Bug #13984</a>: ceph-disk prepare activates the osd on 7.1</i> added</li></ul> Ceph - Bug #15176: partprobe intermittent issues during ceph-disk preparehttps://tracker.ceph.com/issues/15176?journal_id=766592016-08-15T21:56:38ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>