https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2015-04-28T14:48:45ZCeph Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511222015-04-28T14:48:45ZKefu Chaitchaikov@gmail.com
<ul></ul><pre>
common/Throttle.cc: 38: FAILED assert(m >= 0)
ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b]
2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670]
3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd]
4: (FileStore::open_journal()+0xd9) [0x90a3c9]
5: (FileStore::mount()+0x2824) [0x90f9f4]
6: (OSD::init()+0x259) [0x6c47b9]
7: (main()+0x2860) [0x651fc0]
8: (__libc_start_main()+0xf5) [0x7f5d977c4ec5]
9: /usr/bin/ceph-osd() [0x66aff7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre>
<p>looks like the "journal_queue_max_ops" and/or "journal_queue_max_bytes" is negative.</p>
<p>elder one, what are the values of these settings in your ceph cluster? or could you post the output of following commands:</p>
<pre>
ceph-conf --lookup journal_queue_max_bytes
ceph-conf --lookup journal_queue_max_ops
</pre> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511362015-04-28T16:37:38Zelder onetaavi@saarlane.ee
<ul></ul><p>Here it is:</p>
<p><code>ceph-conf -s osd --lookup journal_queue_max_bytes<br />3G<br />ceph-conf -s osd --lookup journal_queue_max_ops<br />5000</code></p>
<p>Is it a "G" suffix causing troubles?</p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511382015-04-28T16:46:14Zelder onetaavi@saarlane.ee
<ul></ul><p>Yeah, seems to be so:</p>
<p><code>ceph --admin-daemon /var/run/ceph/ceph-osd.23.asok config show | grep journal_queue_max_bytes<br /> "journal_queue_max_bytes": "-1073741824"</code></p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511402015-04-28T17:27:13Zelder onetaavi@saarlane.ee
<ul></ul><p>That's strange - changed journal_queue_max_bytes from "3G" to 3221225472 in ceph.conf and restarted all osd's on host but admin socket gives the same negative output:</p>
<p><code>ceph-conf -s osd --lookup journal_queue_max_bytes<br />3221225472<br />ceph --admin-daemon /var/run/ceph/ceph-osd.23.asok config show | grep journal_queue_max_bytes<br /> "journal_queue_max_bytes": "-1073741824",</code></p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511652015-04-29T06:08:50ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>12</i></li><li><strong>Assignee</strong> set to <i>Kefu Chai</i></li><li><strong>Source</strong> changed from <i>other</i> to <i>Community (user)</i></li></ul><p>elder one, yes, this is a bug, because the 3221225472 (i.e. 0xc0000000) is out of range of numeric_limits<int>::max() (i.e. 0x7fffffff on LP64 arch<sup><a href="#fn0">0</a></sup>), and journal_queue_max_bytes is a config whose type is <strong>int</strong>. so your config value of 3221225472 cannot be represented by int.</p>
so we need to:
<ul>
<li>fix the config parsing for int</li>
<li>probably change the type of journal_queue_max_bytes from int to unsigned long long (if we have a good reason to do so)</li>
</ul>
<blockquote>
<p>is it a "G" suffix causing troubles?</p>
</blockquote>
<p>not exactly, you can still use the "G" suffix. but 3G exceeds the limit of <strong>int</strong>. so <strong>2G-1</strong> should be fine.</p>
<p>mean while, you could use "2147483647" instead of 3G for this config as a work around. just out of curiosity, is there any specific reason you need a 3G journal_queue_max_bytes for your system?</p>
<p>---<br />[0] most modern unix platforms on amd64 are LP64. see <a class="external" href="http://www.unix.org/whitepapers/64bit.html">http://www.unix.org/whitepapers/64bit.html</a>. and yes, i believe your linux is not an exception.</p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511662015-04-29T06:12:54ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Subject</strong> changed from <i>OSD-s die after upgrade from Giant to Hammer</i> to <i>OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()</i></li></ul> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511682015-04-29T08:38:23ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>In Progress</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>40</i></li></ul> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511702015-04-29T12:35:24ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Fix Under Review</i></li><li><strong>% Done</strong> changed from <i>40</i> to <i>60</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/4494">https://github.com/ceph/ceph/pull/4494</a></p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511712015-04-29T13:37:24Zelder onetaavi@saarlane.ee
<ul></ul><p>You are right, apparently I don't need 3G journal queue. Value left info config after benchmarking/tuning cluster.<br />That 2G limit I figured out yesterday and now my cluster runs happily on Hammer:)</p>
<p>On Giant, OSD andmin socket gives also negative output, but osd-s start fine.</p>
<p>I propose if any value in config file is out of range, Ceph could use default value for that and warn about that on startup.</p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=511742015-04-29T14:54:01ZKefu Chaitchaikov@gmail.com
<ul></ul><p>elder one wrote:</p>
<blockquote>
<p>I propose if any value in config file is out of range, Ceph could use default value for that and warn about that on startup.</p>
</blockquote>
<p>thank you, elder one, =)</p>
<p>Yeah, i had the same feeling when looking at this issue. will step back and see what we can do to make the error handling of bad config value more user friendly. maybe we should shutdown gracefully at seeing a bad config value when starting up, but reject it at runtime. will file another ticket when i have a better idea.</p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=512552015-05-04T03:38:38ZKefu Chaitchaikov@gmail.com
<ul></ul><p>elder one, i created <a class="issue tracker-2 status-1 priority-4 priority-default" title="Feature: improve OPTION macro to add a check function for invalid options. (New)" href="https://tracker.ceph.com/issues/11525">#11525</a> to track the feature request for improving the invalid option handling.</p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=516522015-05-12T07:09:39ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Resolved</i></li><li><strong>Backport</strong> set to <i>hammer</i></li><li><strong>Regression</strong> set to <i>No</i></li></ul> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=516532015-05-12T07:09:53ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Pending Backport</i></li></ul> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=532162015-06-07T09:05:36ZAbhishek Lekshmananabhishek.lekshmanan@gmail.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/53216/diff?detail_id=51918">diff</a>)</li></ul> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=532302015-06-07T11:22:49ZLoïc Dacharyloic@dachary.org
<ul></ul><p>@abhishek did you intend to edit the summary to include a link to the master pull request <a class="external" href="https://github.com/ceph/ceph/pull/4494">https://github.com/ceph/ceph/pull/4494</a> ? You added <a class="external" href="https://github.com/ceph/ceph/pull/4889">https://github.com/ceph/ceph/pull/4889</a> which is the hammer backport matching the issue <a class="external" href="http://tracker.ceph.com/issues/11738">http://tracker.ceph.com/issues/11738</a>.</p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=532352015-06-07T16:31:30ZAbhishek Lekshmananabhishek.lekshmanan@gmail.com
<ul></ul><p>@loic sorry it was unintended.. Nathan informed me of the mistake I had done; basically if we use the script and feed it some other tracker id; it goes and updates the description to that instead..</p> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=532362015-06-07T16:31:47ZAbhishek Lekshmananabhishek.lekshmanan@gmail.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/53236/diff?detail_id=51943">diff</a>)</li></ul> Ceph - Bug #11484: OPT_INT option interprets "3221225472" as -1073741824, and crashes in Throttle::Throttle()https://tracker.ceph.com/issues/11484?journal_id=576302015-09-02T10:42:14ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>