https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2014-10-27T14:44:40Z
Ceph
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=43717
2014-10-27T14:44:40Z
Samuel Just
sjust@redhat.com
<ul></ul><p>2014-10-25 18:56:23.243456 7fefdcc3e700 20 filestore dbobjectmap: seq is 485<br />2014-10-25 18:56:23.243559 7fefdd43f700 20 filestore dbobjectmap: seq is 486<br />2014-10-25 18:56:23.243602 7fefdd43f700 20 filestore dbobjectmap: seq is 487<br />2014-10-25 18:56:23.243628 7fefdd43f700 20 filestore set_header: setting seq 324<br />2014-10-25 18:56:23.243642 7fefdd43f700 20 filestore set_map_header: setting 485 oid fb2248dc/burnupi5544081-40/45//3/18446744073709551615/2 parent seq 324<br />2014-10-25 18:56:23.243655 7fefdd43f700 20 filestore set_map_header: setting 486 oid fb2248dc/burnupi5544081-40/45//3/484/2 parent seq 324<br />...<br />2014-10-25 18:56:30.801289 7ffbf8345900 0 ceph version 0.86-104-gb05efdd (b05efddb77290b86eb5c150776c761ab84f66f37), process ceph-osd, pid 18817<br />...<br />2014-10-25 18:56:31.104055 7ffbf8345900 20 filestore (init)dbobjectmap: seq is 485<br />...<br />2014-10-25 18:56:40.629704 7ffbe8e4d700 20 filestore(/var/lib/ceph/osd/ceph-2) objectmap clone<br />2014-10-25 18:56:40.629834 7ffbe8e4d700 20 filestore dbobjectmap: seq is 487<br />2014-10-25 18:56:40.629878 7ffbe8e4d700 20 filestore dbobjectmap: seq is 488<br />2014-10-25 18:56:40.629907 7ffbe8e4d700 20 filestore set_header: setting seq 453<br />2014-10-25 18:56:40.629926 7ffbe8e4d700 20 filestore set_map_header: setting 486 oid ac9ea7a1/burnupi5544081-15/head//3/18446744073709551615/1 parent seq 453<br />2014-10-25 18:56:40.629959 7ffbe8e4d700 20 filestore set_map_header: setting 487 oid ac9ea7a1/burnupi5544081-15/64//3/18446744073709551615/1 parent seq 453<br />...<br />2014-10-25 18:56:41.920041 7ffbe8e4d700 20 filestore remove_map_header: removing 486 oid fb2248dc/burnupi5544081-40/45//3/484/2<br />2014-10-25 18:56:41.920056 7ffbe8e4d700 20 filestore clear_header: clearing seq 486<br />...<br />2014-10-25 18:56:41.933829 7ffbe8e4d700 20 filestore remove_map_header: removing 489 oid ac9ea7a1/burnupi5544081-15/head//3/360/1<br />2014-10-25 18:56:41.933842 7ffbe964e700 10 filestore oid: 1113f7/pglog_3.as2/0//-1 not skipping op, *spos 7669.0.4<br />2014-10-25 18:56:41.933844 7ffbe8e4d700 20 filestore clear_header: clearing seq 489<br />2014-10-25 18:56:41.933847 7ffbe964e700 10 filestore > header.spos 0.0.0<br />2014-10-25 18:56:41.933890 7ffbe964e700 15 filestore(/var/lib/ceph/osd/ceph-2) _omap_setkeys meta/1113f7/pglog_3.as2/0//-1<br />2014-10-25 18:56:41.933932 7ffbe964e700 10 filestore oid: 1113f7/pglog_3.as2/0//-1 not skipping op, *spos 7669.0.5<br />2014-10-25 18:56:41.933937 7ffbe964e700 10 filestore > header.spos 0.0.0<br />2014-10-25 18:56:41.934078 7ffbe964e700 10 filestore(/var/lib/ceph/osd/ceph-2) -ERANGE, len is 802<br />2014-10-25 18:56:41.934107 7ffbe964e700 10 filestore(/var/lib/ceph/osd/ceph-2) -ERANGE, got 401<br />2014-10-25 18:56:41.934108 7ffbe8e4d700 20 filestore lookup_parent: parent 486 for seq 489</p>
<p>So, the transaction updating dbobjectmap seq didn't apply for some reason. It would have been submitted before the transaction actually writing out the nodes using it, so that's definitely odd since burnupi5544081-40/45//3/484/2 was apparently written out with seq 486. This must be either a filesystem thing or a leveldb thing. It's not clear to me how it could have been a filesystem thing though.</p>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=43806
2014-10-29T10:00:06Z
Sage Weil
sage@newdream.net
<ul><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>High</i></li></ul><p>does not appear to be a ceph issue.. either bad disk or leveldb corruption or something. lowering priority.</p>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=45096
2014-12-02T13:43:24Z
Samuel Just
sjust@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Rejected</i></li></ul>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=46331
2015-01-12T07:51:37Z
Loïc Dachary
loic@dachary.org
<ul></ul><p><a class="external" href="http://pulpito.ceph.com/loic-2015-01-10_11:54:25-rados-giant-backports-testing-basic-vps/694324/">http://pulpito.ceph.com/loic-2015-01-10_11:54:25-rados-giant-backports-testing-basic-vps/694324/</a><br /><pre>
-6> 2015-01-11 09:18:46.785891 7f6d14685700 1 -- 10.214.140.199:6803/19206 <== osd.5 10.214.138.167:0/6293 12 ==== osd_ping(ping e459 stamp 2015-01-11 09:18:46.774003) v2 ==== 47+0+0 (3607071168 0 0) 0x5d42a00 con 0x5abd000
-5> 2015-01-11 09:18:46.785913 7f6d14685700 1 -- 10.214.140.199:6803/19206 --> 10.214.138.167:0/6293 -- osd_ping(ping_reply e459 stamp 2015-01-11 09:18:46.774003) v2 -- ?+0 0x5a4ec00 con 0x5abd000
-4> 2015-01-11 09:18:46.785929 7f6d14685700 20 osd.2 459 share_map_peer 0x5a45000 already has epoch 459
-3> 2015-01-11 09:18:46.785822 7f6d12e82700 1 -- 10.214.140.199:6802/19206 <== osd.5 10.214.138.167:0/6293 12 ==== osd_ping(ping e459 stamp 2015-01-11 09:18:46.774003) v2 ==== 47+0+0 (3607071168 0 0) 0x5dbfa00 con 0x5abd160
-2> 2015-01-11 09:18:46.785998 7f6d12e82700 1 -- 10.214.140.199:6802/19206 --> 10.214.138.167:0/6293 -- osd_ping(ping_reply e459 stamp 2015-01-11 09:18:46.774003) v2 -- ?+0 0x5499200 con 0x5abd160
-1> 2015-01-11 09:18:46.786011 7f6d12e82700 20 osd.2 459 share_map_peer 0x5a45000 already has epoch 459
0> 2015-01-11 09:18:46.804341 7f6d1c7c3700 -1 os/DBObjectMap.cc: In function 'DBObjectMap::Header DBObjectMap::lookup_parent(DBObjectMap::Header)' thread 7f6d1c7c3700 time 2015-01-11 09:18:46.775993
os/DBObjectMap.cc: 1214: FAILED assert(0)
ceph version 0.87-87-g0ea20e6 (0ea20e6c51208d6710f469454ab3f964bfa7c9d2)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xaa04bf]
2: (DBObjectMap::lookup_parent(std::tr1::shared_ptr<DBObjectMap::_Header>)+0x925) [0x9790e5]
3: (DBObjectMap::_clear(std::tr1::shared_ptr<DBObjectMap::_Header>, std::tr1::shared_ptr<KeyValueDB::TransactionImpl>)+0xe5) [0x9792c5]
4: (DBObjectMap::clear(ghobject_t const&, SequencerPosition const*)+0x1a5) [0x97d325]
5: (FileStore::lfn_unlink(coll_t, ghobject_t const&, SequencerPosition const&, bool)+0x250) [0x8ab400]
6: (FileStore::_remove(coll_t, ghobject_t const&, SequencerPosition const&)+0x172) [0x8ab872]
7: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x3bcc) [0x8bbc5c]
8: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x8be064]
9: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x297) [0x8be317]
10: (ThreadPool::worker(ThreadPool::WorkThread*)+0x48e) [0xa900ee]
11: (ThreadPool::WorkThread::entry()+0x10) [0xa92e80]
12: (()+0x7e9a) [0x7f6d27eabe9a]
13: (clone()+0x6d) [0x7f6d268762ed]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre></p>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=48126
2015-02-16T17:04:55Z
Yuri Weinstein
yweinste@redhat.com
<ul><li><strong>Status</strong> changed from <i>Rejected</i> to <i>New</i></li></ul><p>See in release giant v0.87.1 RC<br />Run: <a class="external" href="http://pulpito.front.sepia.ceph.com/teuthology-2015-02-12_17:29:32-rados-giant-distro-basic-multi/">http://pulpito.front.sepia.ceph.com/teuthology-2015-02-12_17:29:32-rados-giant-distro-basic-multi/</a><br />Job: ['754710']<br />Logs: <a class="external" href="http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-12_17:29:32-rados-giant-distro-basic-multi/754710/">http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-12_17:29:32-rados-giant-distro-basic-multi/754710/</a></p>
<pre>
Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)
ceph version 0.87-159-gccb0914 (ccb0914f76da23acdd7374233cd1939ab80ef3c8)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xb82c4b]
2: (DBObjectMap::lookup_parent(std::tr1::shared_ptr<DBObjectMap::_Header>)+0x798) [0xa2e6d8]
3: (DBObjectMap::_clear(std::tr1::shared_ptr<DBObjectMap::_Header>, std::tr1::shared_ptr<KeyValueDB::TransactionImpl>)+0xf4) [0xa31b24]
4: (DBObjectMap::clear(ghobject_t const&, SequencerPosition const*)+0x202) [0xa36612]
5: (FileStore::lfn_unlink(coll_t, ghobject_t const&, SequencerPosition const&, bool)+0x16b) [0x8ff55b]
6: (FileStore::_remove(coll_t, ghobject_t const&, SequencerPosition const&)+0x8b) [0x8ff9eb]
7: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x2cc1) [0x910441]
8: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x913b64]
9: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x18f) [0x913d0f]
10: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa46) [0xb73dd6]
11: (ThreadPool::WorkThread::entry()+0x10) [0xb74e80]
12: (()+0x8182) [0x7f5c252de182]
13: (clone()+0x6d) [0x7f5c23848efd]
</pre>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=48165
2015-02-16T23:21:45Z
Yuri Weinstein
yweinste@redhat.com
<ul><li><strong>Backport</strong> set to <i>giant</i></li></ul>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=48335
2015-02-20T19:02:29Z
Yuri Weinstein
yweinste@redhat.com
<ul><li><strong>Priority</strong> changed from <i>High</i> to <i>Urgent</i></li></ul>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=48339
2015-02-20T19:54:37Z
Yuri Weinstein
yweinste@redhat.com
<ul></ul><p>Verified that this test:</p>
<p><code>rados/thrash-erasure-code/{clusters/fixed-2.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/default.yaml workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml}</code></p>
<p>passed in run <a class="external" href="http://pulpito.front.sepia.ceph.com/teuthology-2015-02-17_14:43:58-rados-giant-distro-basic-multi/">http://pulpito.front.sepia.ceph.com/teuthology-2015-02-17_14:43:58-rados-giant-distro-basic-multi/</a></p>
<p>Job/logs - <a class="external" href="http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-17_14:43:58-rados-giant-distro-basic-multi/763604/">http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-17_14:43:58-rados-giant-distro-basic-multi/763604/</a></p>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=48342
2015-02-20T21:32:06Z
Samuel Just
sjust@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>7</i></li><li><strong>Assignee</strong> set to <i>Samuel Just</i></li><li><strong>Backport</strong> changed from <i>giant</i> to <i>hammer,giant,firefly,dumpling</i></li></ul><p>2015-02-13 03:40:47.208535 7f0e9b5ec700 20 filestore dbobjectmap: seq is 271<br />2015-02-13 03:40:47.208564 7f0e9adeb700 20 filestore dbobjectmap: seq is 271<br />2015-02-13 03:40:47.208579 7f0e9b5ec700 20 filestore dbobjectmap: seq is 272</p>
<p>Note that there are two threads participating there! DBObjectMap::sync() doesn't take the header_lock!</p>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=48343
2015-02-20T21:32:54Z
Samuel Just
sjust@redhat.com
<ul></ul><p>This can be ignored for the present giant release though.</p>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=48816
2015-03-04T15:12:08Z
Samuel Just
sjust@redhat.com
<ul><li><strong>Status</strong> changed from <i>7</i> to <i>Pending Backport</i></li></ul>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=49145
2015-03-11T09:18:41Z
Loïc Dachary
loic@dachary.org
<ul></ul><ul>
<li>firefly backport <a class="external" href="https://github.com/ceph/ceph/pull/3948">https://github.com/ceph/ceph/pull/3948</a></li>
</ul>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=49498
2015-03-19T09:32:26Z
Loïc Dachary
loic@dachary.org
<ul></ul><ul>
<li>giant backport <a class="external" href="https://github.com/ceph/ceph/pull/4094">https://github.com/ceph/ceph/pull/4094</a></li>
</ul>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=49910
2015-03-26T12:23:21Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>Backport</strong> changed from <i>hammer,giant,firefly,dumpling</i> to <i>hammer,giant,firefly</i></li></ul><p>dumpling is end of life</p>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=49911
2015-03-26T12:27:12Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>Backport</strong> changed from <i>hammer,giant,firefly</i> to <i>giant,firefly</i></li></ul><p>already in hammer</p>
Ceph - Bug #9891: "Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)" in upgrade:firefly-x-giant-distro-basic-multi run
https://tracker.ceph.com/issues/9891?journal_id=49912
2015-03-26T12:27:50Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>