Bug #18599
closedbluestore: full osd will not start. _do_alloc_write failed to reserve 0x10000, etc.
0%
Description
Excited to see how fast I could get into trouble with kraken, I created a small test cluster with 3x 32gb bluestore OSD's.
I set up the cluster by piecing bits of info from different sources, I'm not clear on whether this is was the correct way to set up bluestore, but it did work great until I filled it to the brim with a test file.
the error dump is over 5000 lines so I attached it in a text file.
an oddity of my setup is how ceph-disk prepare allocated my space. On a 32gb disk I ran the following command:
ceph-disk prepare --bluestore /dev/vdb --block.db /dev/vdb --block.wal /dev/vdb
and it produced this partition table:
@Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 34.4GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 106MB 105MB xfs ceph data
3 106MB 21.6GB 21.5GB ceph block.db
4 21.6GB 22.2GB 604MB ceph block.wal
2 22.2GB 34.4GB 12.2GB ceph block@
I noticed the db is way too big, but I rolled with it. My ceph.conf does not define the sizes for any of the partitions because I wanted to see what the defaults would produce. I could not find any guidelines on what size the partitions are supposed to be.
looking at the block partition and seeing that it's 12GB, with 3x OSD's, I expected to run out of space after about 15GB worth of writes (my pool size was 2). and that is what happened.
not surprisingly, 2 of my 3 OSD's will not start. they were the first ones to fill up and at which point there were not enough OSD's up to process further requests.
if this was filestore, I assume that I could manually delete some pg's to get the OSD's to start, but I have no idea how to handle this in bluestore.
one thing that worries me the most is that ceph df reports the following, counting the data, db and wal in the total available space. I would assume that it should only count the data partition in totaling up the space:
2017-01-20 03:12:46.158064 7f89d50d2700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
96272M 65717M 30555M 31.74
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 32603M 0
cephfs_metadata 1 1004k 0 32603M 20
cephfs_data 2 15671M 24.55 32603M 3918
I hope I've given enough info and I'm sorry to bother you if this bug is already known.
Files