Bug #53000
OSDMap/OSDMapTest.BUG_51842/2: ThreadPool::WorkQueue<ParallelPGMapper::Item>::_void_dequeue: pure virtual method called
0%
Description
This failure was reported by Jenkins for pacific branch PR [1], though it does not look like related to that PR, and may be not specific for pacific branch. It was not reproduced after re-running Jenkins. Also I failed to reproduce it locally running the test in loop for a while. May be environment specific.
The test OSDMap/OSDMapTest.BUG_51842/2 aborted with "pure virtual method called" in the thread started by clean_pg_upmaps, in ThreadPool::WorkQueue<ParallelPGMapper::Item>::_void_dequeue().
[ RUN ] OSDMap/OSDMapTest.BUG_51842/2 ID CLASS WEIGHT TYPE NAME -5 3.00000 root infra-1706 -4 1.00000 host host-0 0 1.00000 osd.0 -6 1.00000 host host-1 1 1.00000 osd.1 -7 1.00000 host host-2 2 1.00000 osd.2 -1 0 root default -3 0 rack localrack -2 0 host localhost pure virtual method called terminate called without an active exception ... -2> 2021-10-05T08:33:22.552+0000 7fe076379700 1 BUG_40104::clean_upmap_tp worker finish -1> 2021-10-05T08:33:22.552+0000 7fe079b80700 1 BUG_40104::clean_upmap_tp worker finish 0> 2021-10-05T08:33:22.556+0000 7fe076b7a700 -1 *** Caught signal (Aborted) ** in thread 7fe076b7a700 thread_name:clean_upmap_tp ceph version Development (no_version) pacific (stable) 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_osdmap(+0x2e81e3) [0x557b8f5431e3] 2: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fe086bca3c0] 3: gsignal() 4: abort() 5: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e911) [0x7fe07c0e9911] 6: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa38c) [0x7fe07c0f538c] 7: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3f7) [0x7fe07c0f53f7] 8: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xab155) [0x7fe07c0f6155] 9: (ThreadPool::WorkQueue<ParallelPGMapper::Item>::_void_dequeue()+0x27) [0x557b8f4d9493] 10: (ThreadPool::worker(ThreadPool::WorkThread*)+0x50b) [0x7fe07d8ed7bf] 11: (ThreadPool::WorkThread::entry()+0x36) [0x7fe07d8f1c92] 12: (Thread::entry_wrapper()+0x87) [0x7fe07d8c8e73] 13: (Thread::_entry_func(void*)+0x1c) [0x7fe07d8c8de2] 14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fe086bbe609] 15: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Related issues
History
#1 Updated by Mykola Golub over 2 years ago
Yet another example, now for a PR for the master branch [1, 2], and for OSDMap/OSDMapTest.BUG_51842/1:
[ RUN ] OSDMap/OSDMapTest.BUG_51842/1 ID CLASS WEIGHT TYPE NAME -5 3.00000 root infra-1706 -4 1.00000 host host-0 0 1.00000 osd.0 -6 1.00000 host host-1 1 1.00000 osd.1 -7 1.00000 host host-2 2 1.00000 osd.2 -1 0 root default -3 0 rack localrack -2 0 host localhost { "rule": { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_firstn", "num": 0, "type": "osd" }, { "op": "emit" } ] }, "rule": { "rule_id": 1, "rule_name": "infra-1706", "type": 1, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -5, "item_name": "infra-1706" }, { "op": "chooseleaf_firstn", "num": 3, "type": "host" }, { "op": "emit" } ] } } 2021-10-19T11:21:14.182+0000 7fef7fba9700 1 heartbeat_map reset_timeout 'BUG_40104::clean_upmap_tp thread 0x7fef7fba9700' had timed out after 0.000000000s pure virtual method called terminate called without an active exception *** Caught signal (Aborted) ** in thread 7fef82baf700 thread_name:clean_upmap_tp ceph version Development (no_version) quincy (dev) 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_osdmap() [0x6a7b7b] 2: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fef880073c0] 3: gsignal() 4: abort() 5: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e911) [0x7fef854eb911] 6: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa38c) [0x7fef854f738c] 7: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3f7) [0x7fef854f73f7] 8: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xab155) [0x7fef854f8155] 9: (ThreadPool::WorkQueue<ParallelPGMapper::Item>::_void_dequeue()+0x19) [0x5fea49] 10: (ThreadPool::worker(ThreadPool::WorkThread*)+0x641) [0x7fef86f20841] 11: (ThreadPool::WorkThread::entry()+0x20) [0x7fef86f268c0] 12: (Thread::entry_wrapper()+0x90) [0x7fef86ef6b10] 13: (Thread::_entry_func(void*)+0x18) [0x7fef86ef6a68] 14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fef87ffb609] 15: clone()
[1] https://github.com/ceph/ceph/pull/41323
[2] https://jenkins.ceph.com/job/ceph-pull-requests/84171/console
#2 Updated by Laura Flores over 1 year ago
#3 Updated by Laura Flores over 1 year ago
- File clean_pg_upmaps failure.gz added
#4 Updated by Radoslaw Zarzynski over 1 year ago
- Status changed from New to Fix Under Review
- Pull request ID set to 41323
#5 Updated by Radoslaw Zarzynski over 1 year ago
- Backport set to pacific,quincy
#6 Updated by Radoslaw Zarzynski over 1 year ago
- Status changed from Fix Under Review to Pending Backport
#7 Updated by Backport Bot over 1 year ago
- Copied to Backport #57288: pacific: OSDMap/OSDMapTest.BUG_51842/2: ThreadPool::WorkQueue<ParallelPGMapper::Item>::_void_dequeue: pure virtual method called added
#8 Updated by Backport Bot over 1 year ago
- Copied to Backport #57289: quincy: OSDMap/OSDMapTest.BUG_51842/2: ThreadPool::WorkQueue<ParallelPGMapper::Item>::_void_dequeue: pure virtual method called added
#9 Updated by Backport Bot over 1 year ago
- Tags set to backport_processed
#10 Updated by Mykola Golub over 1 year ago
I don't understand why PR ID for this bug was set to 41323. I mentioned the PR 41323 not as the fix but as an example where this bug was observed when testing that PR.
#11 Updated by Radoslaw Zarzynski over 1 year ago
- Status changed from Pending Backport to New
- Pull request ID deleted (
41323)
Sorry, moving back to New
.
#12 Updated by Kefu Chai over 1 year ago
[ RUN ] OSDMap/OSDMapTest.BUG_51842/2 ID CLASS WEIGHT TYPE NAME -5 3.00000 root infra-1706 -4 1.00000 host host-0 0 1.00000 osd.0 -6 1.00000 host host-1 1 1.00000 osd.1 -7 1.00000 host host-2 2 1.00000 osd.2 -1 0 root default -3 0 rack localrack -2 0 host localhost pure virtual method called ... -2> 2022-09-04T14:17:08.035+0000 7efd9e464640 1 BUG_40104::clean_upmap_tp worker finish -1> 2022-09-04T14:17:08.035+0000 7efd9d462640 1 BUG_40104::clean_upmap_tp worker finish 0> 2022-09-04T14:17:08.219+0000 7efd9ac5d640 -1 *** Caught signal (Aborted) **
spotted on main branch.
#13 Updated by Radoslaw Zarzynski over 1 year ago
- Tags changed from backport_processed to test-failure