https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2018-04-24T17:12:05Z
Ceph
bluestore - Bug #23840: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)
https://tracker.ceph.com/issues/23840?journal_id=111734
2018-04-24T17:12:05Z
Rafal Wadolowski
rwadolowski@cloudferro.com
<ul></ul><p>What is interesting, it looks like the max read of file is limited to 0xffffffff.<br /><pre><code class="c syntaxhl"><span class="CodeRay"> -<span class="integer">11</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.939739</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _replay <span class="hex">0xffffa000</span>: txn(seq <span class="integer">1550847</span> len <span class="hex">0xa6a</span> crc <span class="hex">0x95281bbd</span>)
-<span class="integer">10</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.939745</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _read h <span class="hex">0x55a7876b6f00</span> <span class="hex">0xffffb000</span>~<span class="integer">1000</span> from file(ino <span class="integer">1</span> size <span class="hex">0xffffb000</span> mtime <span class="float">0</span><span class="float">.000000</span> bdev <span class="integer">0</span> extents [<span class="integer">1</span>:<span class="hex">0x15300000</span>+<span class="integer">100000</span>,<span class="integer">0</span>:<span class="hex">0x3d800000</span>+c00000,
-<span class="integer">9</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.939970</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _replay <span class="hex">0xffffb000</span>: txn(seq <span class="integer">1550848</span> len <span class="hex">0xa6a</span> crc <span class="hex">0x86579e18</span>)
-<span class="integer">8</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.939976</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _read h <span class="hex">0x55a7876b6f00</span> <span class="hex">0xffffc000</span>~<span class="integer">1000</span> from file(ino <span class="integer">1</span> size <span class="hex">0xffffc000</span> mtime <span class="float">0</span><span class="float">.000000</span> bdev <span class="integer">0</span> extents [<span class="integer">1</span>:<span class="hex">0x15300000</span>+<span class="integer">100000</span>,<span class="integer">0</span>:<span class="hex">0x3d800000</span>+c00000,
-<span class="integer">7</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.940196</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _replay <span class="hex">0xffffc000</span>: txn(seq <span class="integer">1550849</span> len <span class="hex">0xa6a</span> crc <span class="hex">0x61eaa8e2</span>)
-<span class="integer">6</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.940203</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _read h <span class="hex">0x55a7876b6f00</span> <span class="hex">0xffffd000</span>~<span class="integer">1000</span> from file(ino <span class="integer">1</span> size <span class="hex">0xffffd000</span> mtime <span class="float">0</span><span class="float">.000000</span> bdev <span class="integer">0</span> extents [<span class="integer">1</span>:<span class="hex">0x15300000</span>+<span class="integer">100000</span>,<span class="integer">0</span>:<span class="hex">0x3d800000</span>+c00000,
-<span class="integer">5</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.940427</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _replay <span class="hex">0xffffd000</span>: txn(seq <span class="integer">1550850</span> len <span class="hex">0xa6a</span> crc <span class="hex">0xbcc8bda3</span>)
-<span class="integer">4</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.940433</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _read h <span class="hex">0x55a7876b6f00</span> <span class="hex">0xffffe000</span>~<span class="integer">1000</span> from file(ino <span class="integer">1</span> size <span class="hex">0xffffe000</span> mtime <span class="float">0</span><span class="float">.000000</span> bdev <span class="integer">0</span> extents [<span class="integer">1</span>:<span class="hex">0x15300000</span>+<span class="integer">100000</span>,<span class="integer">0</span>:<span class="hex">0x3d800000</span>+c00000,
-<span class="integer">3</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.940657</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _replay <span class="hex">0xffffe000</span>: txn(seq <span class="integer">1550851</span> len <span class="hex">0xa6a</span> crc <span class="hex">0x8cdd70c6</span>)
-<span class="integer">2</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.940663</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _read h <span class="hex">0x55a7876b6f00</span> <span class="hex">0xfffff000</span>~<span class="integer">1000</span> from file(ino <span class="integer">1</span> size <span class="hex">0xfffff000</span> mtime <span class="float">0</span><span class="float">.000000</span> bdev <span class="integer">0</span> extents [<span class="integer">1</span>:<span class="hex">0x15300000</span>+<span class="integer">100000</span>,<span class="integer">0</span>:<span class="hex">0x3d800000</span>+c00000,
-<span class="integer">1</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.940884</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 <span class="integer">10</span> bluefs _replay <span class="hex">0xfffff000</span>: txn(seq <span class="integer">1550852</span> len <span class="hex">0xa6a</span> crc <span class="hex">0x5e1c1b4f</span>)
<span class="integer">0</span>> <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.943750</span> <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 -<span class="integer">1</span> /build/ceph-<span class="integer">1</span><span class="float">2</span><span class="float">.2</span><span class="float">.4</span>/src/os/bluestore/BlueFS.cc: In function <span class="char">'i</span>nt BlueFS::_replay(<span class="predefined-type">bool</span>)<span class="char">' </span>thread <span class="float">7f</span><span class="integer">9</span><span class="float">5</span>ee81de00 time <span class="integer">2018</span>-<span class="octal">04</span>-<span class="integer">24</span> <span class="integer">16</span>:<span class="integer">18</span>:<span class="integer">3</span><span class="float">3</span><span class="float">.940909</span>
/build/ceph-<span class="integer">1</span><span class="float">2</span><span class="float">.2</span><span class="float">.4</span>/src/os/bluestore/BlueFS.cc: <span class="integer">551</span>: FAILED assert((log_reader->buf.pos & ~super.block_mask()) == <span class="integer">0</span>)
</span></code></pre></p>
<p>In each line of bluefs _read, we see iterating address of log file, and it is breaking at 0xfffff000~1000.</p>
bluestore - Bug #23840: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)
https://tracker.ceph.com/issues/23840?journal_id=111735
2018-04-24T17:17:12Z
Rafal Wadolowski
rwadolowski@cloudferro.com
<ul><li><strong>File</strong> <a href="/attachments/download/3429/shutdown.log">shutdown.log</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/3429/shutdown.log">View</a> added</li></ul><p>Moment of planned shutdown</p>
bluestore - Bug #23840: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)
https://tracker.ceph.com/issues/23840?journal_id=111755
2018-04-24T22:11:43Z
Sage Weil
sage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>7</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/21629">https://github.com/ceph/ceph/pull/21629</a></p>
bluestore - Bug #23840: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)
https://tracker.ceph.com/issues/23840?journal_id=111756
2018-04-24T22:11:56Z
Sage Weil
sage@newdream.net
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>bluestore</i></li><li><strong>Category</strong> deleted (<del><i>OSD</i></del>)</li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Backport</strong> set to <i>luminous</i></li></ul>
bluestore - Bug #23840: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)
https://tracker.ceph.com/issues/23840?journal_id=111788
2018-04-25T10:09:36Z
Rafal Wadolowski
rwadolowski@cloudferro.com
<ul></ul><p>This change is working, I think it could be merge with master.</p>
bluestore - Bug #23840: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)
https://tracker.ceph.com/issues/23840?journal_id=111893
2018-04-26T04:40:53Z
Kefu Chai
tchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>7</i> to <i>Pending Backport</i></li></ul>
bluestore - Bug #23840: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)
https://tracker.ceph.com/issues/23840?journal_id=111897
2018-04-26T06:28:36Z
Enming Zhang
enming.zhang@umcloud.com
<ul></ul><p>I have met the same issue in Luminous.</p>
bluestore - Bug #23840: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)
https://tracker.ceph.com/issues/23840?journal_id=111903
2018-04-26T08:48:19Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/23881">Backport #23881</a>: luminous: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)</i> added</li></ul>
bluestore - Bug #23840: Bluestore OSD hit assert((log_reader->buf.pos & ~super.block_mask()) == 0)
https://tracker.ceph.com/issues/23840?journal_id=112700
2018-05-05T13:48:43Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>