Project

General

Profile

Bug #20645

bluesfs wal failed to allocate (assert(0 == "allocate failed... wtf"))

Added by Zengran Zhang over 6 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

it seems like alloc hint equal end of wal-bdev, but the begin of the wal-bdev is still in use...

my wal-bdev size is 5G, i change the bluefs_compact_log_sync = true, and the osd up again...

error log :
-96> 2017-07-16 11:56:58.312867 7fdb8cd17c80 10 bluefs _flush_and_sync_log allocating more log runway (0xff000 remaining)
-95> 2017-07-16 11:56:58.312869 7fdb8cd17c80 10 bluefs _allocate len 0x400000 from 0
-94> 2017-07-16 11:56:58.312870 7fdb8cd17c80 10 bitmapalloc:reserve instance 94036558826352 num_used 752 total 5120
-93> 2017-07-16 11:56:58.312872 7fdb8cd17c80 10 bitmapalloc:allocate instance 94036558826352 want_size 4194304 alloc_unit 1048576 hint 5368709120
-92> 2017-07-16 11:56:58.312879 7fdb8cd17c80 -1 bluefs _allocate allocate failed on 0x400000 min_alloc_size 0x100000
-91> 2017-07-16 11:56:58.312879 7fdb8cd17c80 0 bitmapalloc:dump instance 0x558697af4b70
-90> 2017-07-16 11:56:58.312881 7fdb8cd17c80 0 bitalloc:dump_state zone 0 dump start
-89> 2017-07-16 11:56:58.312881 7fdb8cd17c80 0 bitalloc:0:: 0xffffffffffffffff
-88> 2017-07-16 11:56:58.312882 7fdb8cd17c80 0 bitalloc:1:: 0xffffffffffffffff
-87> 2017-07-16 11:56:58.312882 7fdb8cd17c80 0 bitalloc:2:: 0xffffffffffffffff
-86> 2017-07-16 11:56:58.312883 7fdb8cd17c80 0 bitalloc:3:: 0xffffffffffffffff
-85> 2017-07-16 11:56:58.312883 7fdb8cd17c80 0 bitalloc:4:: 0xffffffffffffffff
-84> 2017-07-16 11:56:58.312884 7fdb8cd17c80 0 bitalloc:5:: 0xffffffffffffffff
-83> 2017-07-16 11:56:58.312884 7fdb8cd17c80 0 bitalloc:6:: 0xffffffffffffffff
-82> 2017-07-16 11:56:58.312884 7fdb8cd17c80 0 bitalloc:7:: 0xf000000000000000
-81> 2017-07-16 11:56:58.312885 7fdb8cd17c80 0 bitalloc:8:: 0x0
-80> 2017-07-16 11:56:58.312885 7fdb8cd17c80 0 bitalloc:9:: 0x0
-79> 2017-07-16 11:56:58.312885 7fdb8cd17c80 0 bitalloc:10:: 0x0
-78> 2017-07-16 11:56:58.312886 7fdb8cd17c80 0 bitalloc:11:: 0x0
-77> 2017-07-16 11:56:58.312886 7fdb8cd17c80 0 bitalloc:12:: 0x0
-76> 2017-07-16 11:56:58.312887 7fdb8cd17c80 0 bitalloc:13:: 0x0
-75> 2017-07-16 11:56:58.312887 7fdb8cd17c80 0 bitalloc:14:: 0x0
-74> 2017-07-16 11:56:58.312887 7fdb8cd17c80 0 bitalloc:15:: 0x0
-73> 2017-07-16 11:56:58.312888 7fdb8cd17c80 0 bitalloc:dump_state zone 0 dump end
-72> 2017-07-16 11:56:58.312889 7fdb8cd17c80 0 bitalloc:dump_state zone 1 dump start
-71> 2017-07-16 11:56:58.312889 7fdb8cd17c80 0 bitalloc:0:: 0x0
-70> 2017-07-16 11:56:58.312889 7fdb8cd17c80 0 bitalloc:1:: 0x0
-69> 2017-07-16 11:56:58.312890 7fdb8cd17c80 0 bitalloc:2:: 0x0
-68> 2017-07-16 11:56:58.312890 7fdb8cd17c80 0 bitalloc:3:: 0x0
-67> 2017-07-16 11:56:58.312890 7fdb8cd17c80 0 bitalloc:4:: 0x0
-66> 2017-07-16 11:56:58.312891 7fdb8cd17c80 0 bitalloc:5:: 0x0
-65> 2017-07-16 11:56:58.312891 7fdb8cd17c80 0 bitalloc:6:: 0x0
-64> 2017-07-16 11:56:58.312892 7fdb8cd17c80 0 bitalloc:7:: 0x0
-63> 2017-07-16 11:56:58.312892 7fdb8cd17c80 0 bitalloc:8:: 0x0
-62> 2017-07-16 11:56:58.312892 7fdb8cd17c80 0 bitalloc:9:: 0x0
-61> 2017-07-16 11:56:58.312893 7fdb8cd17c80 0 bitalloc:10:: 0x0
-60> 2017-07-16 11:56:58.312893 7fdb8cd17c80 0 bitalloc:11:: 0x0
-59> 2017-07-16 11:56:58.312893 7fdb8cd17c80 0 bitalloc:12:: 0x0
-58> 2017-07-16 11:56:58.312894 7fdb8cd17c80 0 bitalloc:13:: 0x0
-57> 2017-07-16 11:56:58.312894 7fdb8cd17c80 0 bitalloc:14:: 0x0
-56> 2017-07-16 11:56:58.312895 7fdb8cd17c80 0 bitalloc:15:: 0x0
-55> 2017-07-16 11:56:58.312895 7fdb8cd17c80 0 bitalloc:dump_state zone 1 dump end
-54> 2017-07-16 11:56:58.312895 7fdb8cd17c80 0 bitalloc:dump_state zone 2 dump start
-53> 2017-07-16 11:56:58.312896 7fdb8cd17c80 0 bitalloc:0:: 0x0
-52> 2017-07-16 11:56:58.312896 7fdb8cd17c80 0 bitalloc:1:: 0x0
-51> 2017-07-16 11:56:58.312896 7fdb8cd17c80 0 bitalloc:2:: 0x0
-50> 2017-07-16 11:56:58.312897 7fdb8cd17c80 0 bitalloc:3:: 0x0
-49> 2017-07-16 11:56:58.312897 7fdb8cd17c80 0 bitalloc:4:: 0x0
-48> 2017-07-16 11:56:58.312898 7fdb8cd17c80 0 bitalloc:5:: 0x0
-47> 2017-07-16 11:56:58.312898 7fdb8cd17c80 0 bitalloc:6:: 0x0
-46> 2017-07-16 11:56:58.312898 7fdb8cd17c80 0 bitalloc:7:: 0x0
-45> 2017-07-16 11:56:58.312899 7fdb8cd17c80 0 bitalloc:8:: 0x0
-44> 2017-07-16 11:56:58.312899 7fdb8cd17c80 0 bitalloc:9:: 0x0
-43> 2017-07-16 11:56:58.312899 7fdb8cd17c80 0 bitalloc:10:: 0x0
-42> 2017-07-16 11:56:58.312900 7fdb8cd17c80 0 bitalloc:11:: 0x0
-41> 2017-07-16 11:56:58.312900 7fdb8cd17c80 0 bitalloc:12:: 0x0
-40> 2017-07-16 11:56:58.312900 7fdb8cd17c80 0 bitalloc:13:: 0x0
-39> 2017-07-16 11:56:58.312901 7fdb8cd17c80 0 bitalloc:14:: 0x0
-38> 2017-07-16 11:56:58.312901 7fdb8cd17c80 0 bitalloc:15:: 0x0
-37> 2017-07-16 11:56:58.312902 7fdb8cd17c80 0 bitalloc:dump_state zone 2 dump end
-36> 2017-07-16 11:56:58.312902 7fdb8cd17c80 0 bitalloc:dump_state zone 3 dump start
-35> 2017-07-16 11:56:58.312902 7fdb8cd17c80 0 bitalloc:0:: 0x0
-34> 2017-07-16 11:56:58.312903 7fdb8cd17c80 0 bitalloc:1:: 0x0
-33> 2017-07-16 11:56:58.312903 7fdb8cd17c80 0 bitalloc:2:: 0x0
-32> 2017-07-16 11:56:58.312903 7fdb8cd17c80 0 bitalloc:3:: 0x0
-31> 2017-07-16 11:56:58.312904 7fdb8cd17c80 0 bitalloc:4:: 0x0
-30> 2017-07-16 11:56:58.312904 7fdb8cd17c80 0 bitalloc:5:: 0x0
-29> 2017-07-16 11:56:58.312904 7fdb8cd17c80 0 bitalloc:6:: 0x0
-28> 2017-07-16 11:56:58.312905 7fdb8cd17c80 0 bitalloc:7:: 0x0
-27> 2017-07-16 11:56:58.312905 7fdb8cd17c80 0 bitalloc:8:: 0x0
-26> 2017-07-16 11:56:58.312906 7fdb8cd17c80 0 bitalloc:9:: 0x0
-25> 2017-07-16 11:56:58.312906 7fdb8cd17c80 0 bitalloc:10:: 0x0
-24> 2017-07-16 11:56:58.312906 7fdb8cd17c80 0 bitalloc:11:: 0x0
-23> 2017-07-16 11:56:58.312907 7fdb8cd17c80 0 bitalloc:12:: 0x0
-22> 2017-07-16 11:56:58.312907 7fdb8cd17c80 0 bitalloc:13:: 0x0
-21> 2017-07-16 11:56:58.312907 7fdb8cd17c80 0 bitalloc:14:: 0x0
-20> 2017-07-16 11:56:58.312908 7fdb8cd17c80 0 bitalloc:15:: 0x0
-19> 2017-07-16 11:56:58.312908 7fdb8cd17c80 0 bitalloc:dump_state zone 3 dump end
-18> 2017-07-16 11:56:58.312909 7fdb8cd17c80 0 bitalloc:dump_state zone 4 dump start
-17> 2017-07-16 11:56:58.312909 7fdb8cd17c80 0 bitalloc:0:: 0x0
-16> 2017-07-16 11:56:58.312909 7fdb8cd17c80 0 bitalloc:1:: 0x0
-15> 2017-07-16 11:56:58.312910 7fdb8cd17c80 0 bitalloc:2:: 0x0
-14> 2017-07-16 11:56:58.312910 7fdb8cd17c80 0 bitalloc:3:: 0x0
-13> 2017-07-16 11:56:58.312910 7fdb8cd17c80 0 bitalloc:4:: 0x0
-12> 2017-07-16 11:56:58.312911 7fdb8cd17c80 0 bitalloc:5:: 0x0
-11> 2017-07-16 11:56:58.312911 7fdb8cd17c80 0 bitalloc:6:: 0x0
-10> 2017-07-16 11:56:58.312911 7fdb8cd17c80 0 bitalloc:7:: 0x0
-9> 2017-07-16 11:56:58.312912 7fdb8cd17c80 0 bitalloc:8:: 0x0
-8> 2017-07-16 11:56:58.312912 7fdb8cd17c80 0 bitalloc:9:: 0x0
-7> 2017-07-16 11:56:58.312913 7fdb8cd17c80 0 bitalloc:10:: 0x0
-6> 2017-07-16 11:56:58.312913 7fdb8cd17c80 0 bitalloc:11:: 0xfffffffffff
-5> 2017-07-16 11:56:58.312913 7fdb8cd17c80 0 bitalloc:12:: 0xffffffffffffffff
-4> 2017-07-16 11:56:58.312914 7fdb8cd17c80 0 bitalloc:13:: 0xffffffffffffffff
-3> 2017-07-16 11:56:58.312914 7fdb8cd17c80 0 bitalloc:14:: 0xffffffffffffffff
-2> 2017-07-16 11:56:58.312914 7fdb8cd17c80 0 bitalloc:15:: 0xffffffffffffffff
-1> 2017-07-16 11:56:58.312915 7fdb8cd17c80 0 bitalloc:dump_state zone 4 dump end
0> 2017-07-16 11:56:58.328925 7fdb8cd17c80 -1 /tmp/release/Ubuntu/WORKDIR/ceph-12.0.2-20-g5f75d5c/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_allocate(uint8_t, uint64_t, mempool::bluefs::vector<bluefs_extent_t>*)' thread 7fdb8cd17c80 time 2017-07-16 11:56:58.312916
/tmp/release/Ubuntu/WORKDIR/ceph-12.0.2-20-g5f75d5c/src/os/bluestore/BlueFS.cc: 1823: FAILED assert(0 == "allocate failed... wtf")

History

#1 Updated by Sage Weil over 6 years ago

  • Status changed from New to Closed

can you retset on current master? this is pretty old code. please reopen if the bug is still present.

#2 Updated by Zengran Zhang over 6 years ago

sorry for not post the version, the assert occured in v12.0.2. maybe its similar with #18054, but i think they are different...not sure i t could reproduce soon...

#3 Updated by Sage Weil over 6 years ago

  • Status changed from Closed to 12

#4 Updated by Igor Fedotov over 5 years ago

  • Status changed from 12 to Closed

Also available in: Atom PDF